Lecture 20: Reed-Solomon Codes

Key Takeaways

Okay. So I want to start with a timeline. So Shannon 1948 discovered um the noisy coding theorem. um Hamming discovered um Hamming codes in 1950. Um Reed and Solomon discovered Reed Solomon codes in 1960. Um, concatenated codes were discovered in 1966. and Burleamp Massie decoding algorithm for read Solomon codes was discovered in 1969. So this is a lot of steps and it turns out that the burlac massi decoding algorithm for read solomman codes along with concatenated codes was what made um using eric recting codes practical. And in Voyager deep space was 1977 and one of the Voyager aircrafts is still transmitting signals to Earth and they're doing that using Reed Solomon codes which as I'm going to which is what I'm going to tell you about later in the lecture. And after all this stuff, Reed Solomon codes were still used until um oh, I don't know around 19 mid 1990s. Actually, they were still used until probably around 2000. But in the mid 1990s, these three French engineers. So read Solomon codes do not actually meet Shannon's bound from the noisy coding theorem. Reed Solomon codes come within a factor within a constant factor of it. But you know between 1948 for the next 40some years coding theorist has struggled to find a technique for actually meeting Shannon's bound and in 199 something this paper appeared by three French engineers. I mean, they weren't even they weren't even coding theorists. They were three people at a at a French telecom company and they claimed to meet Shannon's bound with a completely different technique and none of the coding theorist researchers believed it until they programmed up and it worked and even that it took many years for them to explain why it worked and in fact they still don't understand why it worked, why turbo codes work. They do understand why uh similar type of codes called LDP C codes work really well. But um so now um you know after this discovery read Solomon codes gradually faded out of use in new applications and turbo codes and LDPC codes and polar codes which were discovered in the 2000s took over. So, but I, you know, I find it kind of amazing that you needed something like three discoveries to actually make Shannon's Thurm practical. So, what is a Reed Solomon code? Okay. Well, it's based on a ancient theorem from mathematics uh polomial of degree D over a finite field. has at most d roots. [sighs] Okay. So a read Solomon code you have a message M0 through MK well MK minus one because we're starting with zero. Associate the polomial P of X = M0 + M1 X + M2 X^2 + M K -1 X K -1. So this is the message and once you have the message you construct this polomial although um I mean we're not going to actually um transmit the polinomial what we're going to transmit is um the code which is P of 0, P of 1, P of 2, all the way up through P of N minus one. Now you know to do this we need n is less than or equal to okay so this is over a finite field and for this lecture we will assume the finite field is the integers mod p in real life um read mo read sol read read solomon codes were actually they actually use the polomial Z over the finite field with 2 to the K elements but that's because they're using binary and 2 to the K is very convenient for binary but the theory is exactly the same if we take Z subp and um so for this well we need n minus one different elements in the field and are n different elements in the field and for that we need n is is at most p and in fact as an aside you don't have to use 0 1 2 through n minus one for the um for these points points. You could just use any n distinct points, but you know, the lecture notes start out by saying choose end points and then they say later we'll use the points 0 through n minus one. So I'm just going to start out with 0 through n minus one. Okay. and note read Solomon's codes are linear and the generator matrix G is equal to 1 0 0 0 0 1 1 1 2 4 um 8 etc. 2 to the k -1 3 9 27 etc. and 3 to the k minus one all the way up to well how many um how many values of the polomial are there are I mean how many numbers do you plug into the polomial there are n minus one so this last row last column is one n minus 1 n -1^ 2ar through n -1 to the k -1 and I should note that these numbers are all mod z subp. So if P was 13, this entry would not be 27 but one. Oh well, I mean 27 is one mod 13 but [snorts] yeah. So this is the generary matrix G. And if you put in m0 m1 through mk, the i entry in the code word is just 1 + m1 * 3 1st + m2 * 3 2 + m3 * 3 cubed etc. Which is the value of p of x at 3. So they're linear codes which means that um we want to find how many errors they can correct. So remember okay so need to find minimum weight non zero code If weight let's give this a name C. If weight of C is equal to D we can correct D -1 / 2 errors. thermum and that's because the minimum distance between any two code words is d minus one / two minimum Wait. Nonzero code word has weight n - k + 1. proof the code word. Well, we know what the code word is. It's just P of 0, P of 1 through P of N minus one. where P = M0 + M1 X + M K -1 X to the K -1 is the is a polomial of degree K and remember that the polomial this polomial is actually the coefficients are the message so it could be any polinomial of degree K of degree K minus one sorry and now we use this theorem that I reminded you a A little while ago, a polomial of degree D has at most d roots. So P of X has at most k minus one zeros. So at most k minus one of these entries are zero, which means at least n minus k + one um values of p subi are non zero because you have n values in the code word k minus one of them or fewer are zero which means this many have to be non zero. So the minimum Hamming weight is n - k + 1. Suppose we want to correct um e errors. Well, we need minimum distance to be greater than equal to 2 e + 1. And that means we need n - k + 1 greater than or equal to 2 e + 1 or n minus k is greater than equal to 2 e. Okay. So that are our read solomon codes. Now remember that to have an have a usable code, we not only need to be able to encode, but we also need to be able to correct errors and decode. So the question is how do you decode? So this is what I will spend probably the next half hour telling you. Okay. And I am not going to give you the not going to give you the burlac massie decoding algorithm because that is conceptually very or I shouldn't say very that's conceptually fairly difficult and I'm going to give you a much easier decoding algorithm to explain which is not as efficient as a burleamp Massie decoding algorithm and there's another decoding algorithm which is probably equally efficient which is based on the uklidian algorithm for finding you know greatest common divisor but I'm not going to give you that one either because again that's fairly difficult conceptually How do we decode? So this decoding algorithm is based on a theorem. So the suppose we have a code word C equals P of 0 P of 1 through P of N minus one we get the transmission H rather we receive the thing the um code word with some errors in it. C tilda equals R comma R1 comma R2 through R NUS1. And these guys are not going to have a polomial where RA equals P of I. For some of these, we have RA equals P of I. But for some others, we won't. Assume there are L less than E errors. Then there are positions I1 I2 through IIL where P of I subL is not equal to R. P sub I sub K is not equal to R of R sub I subK. So we have at most L errors and we assume there are positions in positions I1 through IIL and L is less than or equal to E because we're assuming there are fewer errors than the code is able to decode. So this is the hypothesis of the theorem and I need to put the theorem itself on a different board are the then um continued There are nonzero polomials f and q [clears throat] and q of x of degree less than or equal K + E - 1 such that Q of I is equal to R subi * F sub I. Okay. So this looks like a totally crazy theorem. What does it have to do with decoding? Well, um I mean you will see what have to do with decoding when I give you the decoding algorithm. But first let's prove the theorem. Let f ofx equal x - i1 x - i2 through x - i subl is equal to the product of x - i subk k = 1 through l. So this is you just take all the positions which are in error. Of course you don't know which positions are in error when you receive this this word which is you know the code word with at most e errors. [snorts] But we'll let f ofx be that because and um and we will let q of x = p of x * f ofx. Okay. Oh, I think we If I I should have put this down. Q of I= R subi F subi for I = 0 1 through N minus one. So this is only true at the positions which give you values in the code. And of course this you know r subi is not defined for any position other than 0 1 through n minus one. So this condition is probably a little bit unnecessary but still I'm going to write it. So if I is in I1 through I sub L. Then f of i equals zero. Right? Because f ofx was just the product of x minus i sub k where i is in this set and q of x = p of x f ofx = z or q of i = p of I * f of i equals z. So we wanted to show that q of i = r sub i * f of i. Q of I = R subi * F of I equals Z because remember Q of X was P of X * F ofX. Okay. So the other alternative is if I is not in the set of positions where things are in error then well if there are no errors then r subi equals p of i and q of I is equal to P of I * R sub I um Q of I equals R P Q of I= P of I * F of I which is equal to since RA I equals P of I it's R subi * F of I and that's what we wanted Q of I= R subi* F of I. So whether or not I is in the position containing the errors Q of I= P of I * F of I. Okay, so that's the theorem. We have these um two polomials f ofx and q of x which have this relation between them. How do we decode linear algebra? So the lecture notes give you a explicit example of how to decode but I am not going to go through it in class because you know it's really fairly tedious to um go through and I will just explain the theory. So we have Q of I = R subi * F of I. So this is a whole bunch of equations. Q of 0 = R0 * F of 0. Q of 1 = R1 * F of 1. Q of 2 = R2 * F of 2 all the way down to Q of N -1 = R sub N -1 * F N -1. So these equations you can think of these equations as linear equations and the coefficients of the polinomials. And okay, so what do they look like in terms of the linear equations in the um well Q of 0? Well, I should probably say what the coefficients are. Q of X = Q0 + Q1 X + Q sub um K + C - 1 X to the K + E - one and F of X = F0 plus F_sub_1X plus so this has degree e. Um so it's f sub e um x to the e. Okay. So what do these equations look like? Well, the first one q of 0 is just q0. So it's Q0 equals R0 F0. The next one is Q of 1. So it's just Q0 plus Q1 + Q2 plus dot dot dot plus Q. And what was the degree? K + E minus one is equal to R1 F0 + F1 plus D plus F sub. And the next one is F of 2. So it's q0 + 4 + 2 q1 + 4 q2 + dot dot dot + 2 to the k + e minus one q sub k + e minus one. Of course this is mod p. So you don't so the numbers don't actually grow huge. And this equals um R sub 2 F0 + 2 F1 plus dot dot dot plus 2 to the E F sub. Okay, I'm going to claim this is a linear equation in coefficients of Q and R. Well, what are the coefficients of Q1 and R? Well, there's Q0, Q1, Q2, Q plus C minus one, and F0, F_sub_1 through F sub. And it's clearly a linear equation in the coefficients. So linear algebra. Okay. We have a bunch of linear equations and a bunch of variables. How many solutions do they have? Well, Let's see how many variables. Well, there are e + one variables in f ofx because they're e+1 coefficients. And there are k + e minus one + one variables in q of x because there are k plus e coefficients. So a variable, how many variables there are? K plus E um plus E + 1. How many equations? Well, there are n equations because we have one for q0, q1 all the way up to q of n minus one. n equations. But somewhere we have n minus k equals 2 e. And up there we said we needed n minus k greater than or equal to 2 e to correct the errors. But let's assume that n minus k equals 2 e because that's the minimum that's the minimum we need to correct e errors. So n minus k = 2 e. So we have k + 2 e + 1 variables and k + 2 e equations. And these equations are all linearly independent. And how do you see that? Okay, I'm not going to really explain this in full detail. But remember G is equal to 1 0 0 0 1 1 1 through 1 1 2 4 8 through 2 to the um n minus one 3 9 27 through 3 to the nus1 etc. So this is a Vandermon matrix and Vandermon matrices have full rank. And this is a theorem which we could easily have proved at some point in our course but we didn't and I'm not going to prove it today. But this is the generating function. This is sorry the generating ma matrix for our code and it has full rank. And by using that fact and a similar fact on the right hand side for the fs, you can show that these linear equations have full rank. Which means that um we have n minus k equals we have k + 2 e um linearly independent equations and k + 2 e + 1 variables and that means there exist there exists a nonzero So solution and in fact um you can use the fact that f sub equals 1 because f was yeah f was this and so f sub equals 1. Use f subb = 1 to get a unique solution assuming l equals e. Okay, I don't know what happens when l is the number of actual errors is smaller than e. the maximum number of errors you can correct. Um, and other error correction algorithms for um, read Solomon codes, you basically have to go through all possibilities of how big L is and check each one out. Except there are usually much simpler ways to do that than going through all of these um, possibilities. [clears throat] Okay, so we have one solution to all of these equations. And that means we can figure out what q of x and f ofx are because we know that q of x and sorry q of x and p of x are because we know that q of x and p of x satisfy these um I'm sorry we know we can figure out what f ofx and q ofx are because we had um linear equations for the coefficients of q of x and f ofx. So we can deduce q ofx and f ofx. So linear algebra gives q of x and f ofx and we had p of x this was the message * f ofx = q of x. So p of x = q ofx over f ofx and we have gotten the message back. Okay, so that's how to decode read Solomon codes. The last thing I want to do is tell you about concatenated codes and what and why these make read Solomon codes truly practical. So we have n = k + e plus 2 e sorry for read solen codes and let's assume Assume the Reed Solomon codes over the finite field with 2 to the eth elements. So I don't know whether professor Mitra mentioned it before but there's a finite field for every power of a prime and no one of the very common finite fields used for read Solomon codes in practice is 2 to the eth. So let's say n = 256 that's 2 8 k = 150 and that implies that e = n - k / 2 because we had um n minus k= 2 And 256 - 150 / 2 is 53 can encode let's see K was 150 150 bytes if we assume a bite is eight bits into 256 bytes Correct. 53 errors, which means 53 is the number of bytes that can have errors in them. And if you think about this in general cannot correct an error rate greater than n / two because the number of errors you can correct is n minus k / 2 And even if you use the smallest possible code which encodes a single bite that's eight bits um you cannot correct more than any of our two errors. Okay. Let's assume the bit error rate is Um [sighs] 10%. So you're transmitting zeros and ones and you're um grouping them into blocks of eight bits, but each of your bit is likely to be have a 10% error. Probability a bite that's a group of eight bits is in error is 1 - 1 - 110th to the 8th. Okay. which equals 57%. I mean this is the probability that each of your bits is not an error. So you raise it to the eth. That's the probability that all of your bits do not have an error. And the probability of an error is one minus this. That's 57%. So if you're trying to use a read Solomon code with blocks of size eight, um you cannot find parameters that will let it correct the error. I want to say what do you do? You use concatenated codes. You can look in code tables. And these are available on the web. Find that there is uh 19 8 seven code. This encodes eight bits into 19 bits and the distance is equal to seven which means it corrects three errors. Okay. So, we should call that code something. So, why don't we call it C inner? Okay. and use read Solomon code and somewhere I had this example of 150 bytes into 256 bytes and corrects 53 errors. So I guess that is the 250 that's n and 150 and um corrects 53 errors because the distance is 7. So how do you canatenate two codes? So the first thing you do is apply the outer code to your message. Wait. Yeah. What does that look like? Well, your message you break into blocks of of eight bits and eight eight bits can be represented as numbers between 1 and 25 between 0 and 256. So your message might look like um 317, 251, 109, 113, 75, etc. And it has 150 of these 150 of these things. So now you encode it in your read Solomon code using your read Solomon code and you get 250 code word length 250 looks pretty much the same 113 11 17 257 etc. And there are 250 of these numbers. Take each of these numbers code using C inner. And now to decode you first decode CN and you get this code word and now you decode it using the code the um decoding algorithm for read Solomon codes and you get your message. And you have taken your message um take I guess 8 * 150 is equal to um I think 1,200 bits. N2 what was our 1987 bits codes into 19 * 256 which okay I did not do this multiplication before I before class so 250 * 20 is um is around 5,000 bits. So you've taken 1,200 bits to 5,000 bits. You have um and you can ask what is the probability you decode successfully with a 10% bit error rate. Okay, so I used um Mathematica for this but at a 10% error rate um if you have Eight bits 10% error rate probability of greater than equal to three errors is 11%. And now the chance and what did we have? We have an 11% error rate in the bytes. We have 256 bytes. And if there are fewer than 53 of them in error, you can decode with high probability. And you know how with from probability theory, you know how to figure out the probability that you cannot decode. But the probability you cannot decode is approximately 10 to the -60th which is really incredibly small. So that means that you can you know if you take a canatinated code and use this code for the outer code and this code for the inner code and you have a 10% error rate on your bits, your information almost always gets through completely intact. And actually so this you know 10% error rate um we've taken 1 1200 bits to 5,000 bits we've multiplied the number of bits by four. Now if you look at Shannon's theorem if you have 1 1200 bits and a 10% error rate you should be able to get by with multiplying the number of bits by two. So the ideal code would give us 2400 bits and we've used twice that many 5,000 bits. So that's not that horrible. I mean if we use turbo codes or LDBC codes or um polar codes which were invented much later than Reed Solomon codes, we would reduce that to something very close to the to Shannon's bound. But you know this was good enough for you know let's see [snorts] yeah so this was good enough for 30 years I mean the um all the machinery for using read Solomon codes had been discovered by 1970 and they remained in widespread use well they remain in widespread use Today, anytime you play a CD, of course, very few people are playing CDs anymore, but anytime you play a CD, you're using Reed Sullivan codes. Okay, [snorts] there's one last thing I want to say. Um, well, I've told you how to decode the outer code. How can you decode the inner code? I mean, it's a 198 seven code. There are 2 to the 19th which is approximately um 500,000 code words and can use table lookup for these. You have your computer have in your computer a complete table of all now there are 500,000 um possible yeah there are only 256 code words but those get mapped into 19bit um words And there are 500,000 19 bit words. And you can use table lookup to decode. You can just list all possible 500,000 words you might receive. And to decode them, you look up in the table to find the nearest code word. And that's the decoding. And nowadays, this is incredibly easy. In 1977 when Voyager did it, I suspect that the Voyager spacecraft did not have enough memory for 500,000 table lookup words. So, they must have done something cleverer. Well, one thing you could do is instead of using um instead of using uh read Solomon codes with eight bits in each block, use ones with smaller number of bits in each block and then you will get a smaller um code word. Or another thing you could do is for the inner code you could use a Hamming code words and Hamming Hamming code and Hamming codes are very easy to decode. although they really do not get quite as good performance as you know three error correcting codes and there are other things you can do too so I don't actually know what they did for Voyager but I um it's probably somewhere on the line and I could look it up okay so that's all I have on Eric recting codes and um in fact this is my last lecture of the course because professor Mitra will be giving the next two non um non-exam lectures and we have one exam for you. Okay.