Welcome to GOFASTMOTORSPORTS.com ... ... ... Green Flag - Green Flag - Go Go Go ... ... Brought to you by Ricks Satellite - Where the Big Dish Rules!

Ricks Satellite Wildfeed and Backhaul Forum
Register Latest Topics
 
 
 


Reply
  Author   Comment   Page 3 of 21      Prev   1   2   3   4   5   6   Next   »
midwestmac

Registered:
Posts: 2,275
Reply with quote  #31 
Quote:
Originally Posted by mips
[

Here's an example from the streamg18friday818pm.ts log file:
BBF @ 51617527    MATYPE=$4200 [GS-C  MIS ACM ISSYI=0 GLT=0 RO=2 ISI=$00] UPL=$0000 DFL=$2938 SYNC=$B1 SYNCD=$0000 CRC1=$82
BBF @ 51618856    MATYPE=$4200 [GS-C  MIS ACM ISSYI=0 GLT=0 RO=2 ISI=$00] UPL=$0000 DFL=$2938 SYNC=$B2 SYNCD=$0000 CRC1=$D2
BBF @ 51620185    MATYPE=$9744 [GS-HE MIS CCM ISSYI=0 GLT=1 RO=3 ISI=$44] UPL=$3850 DFL=$4B34 SYNC=$D9 SYNCD=$CC0E CRC1=$1F
BBF @ 51622601    MATYPE=$CE01 [TS    MIS ACM ISSYI=1 NPD=1 RO=2 ISI=$01] UPL=$0D51 DFL=$E4F0 SYNC=$85 SYNCD=$40F8 CRC1=$64
BBF @ 51629937    MATYPE=$4200 [GS-C  MIS ACM ISSYI=0 GLT=0 RO=2 ISI=$00] UPL=$0000 DFL=$2938 SYNC=$20 SYNCD=$0000 CRC1=$D0
BBF @ 51631266    MATYPE=$4200 [GS-C  MIS ACM ISSYI=0 GLT=0 RO=2 ISI=$00] UPL=$0000 DFL=$2938 SYNC=$21 SYNCD=$0000 CRC1=$53



Mips I was trying to figure out were the MATYPE 9744 and CE01 came from if you get a chance can you look at that again? I looked at it and again and I get the same log file but looking at it with a hex editor it looks different than what your app shows it looks like there's some 42's it missed.

I found this one too "MATYPE=$CE01" am I wrong? but the log doesn't look right.
Thanks




hexofg18.JPG


__________________
Azbox Ultra, Pansat 2500, Prof7301,Tbs 6925,5980, Genpix 8psk card, Dektec 2137c, Hauppauge 950q
mips

Registered:
Posts: 529
Reply with quote  #32 
Quote:
Originally Posted by midwestmac
Quote:
Originally Posted by mips
[

Here's an example from the streamg18friday818pm.ts log file:
BBF @ 51617527    MATYPE=$4200 [GS-C  MIS ACM ISSYI=0 GLT=0 RO=2 ISI=$00] UPL=$0000 DFL=$2938 SYNC=$B1 SYNCD=$0000 CRC1=$82
BBF @ 51618856    MATYPE=$4200 [GS-C  MIS ACM ISSYI=0 GLT=0 RO=2 ISI=$00] UPL=$0000 DFL=$2938 SYNC=$B2 SYNCD=$0000 CRC1=$D2
BBF @ 51620185    MATYPE=$9744 [GS-HE MIS CCM ISSYI=0 GLT=1 RO=3 ISI=$44] UPL=$3850 DFL=$4B34 SYNC=$D9 SYNCD=$CC0E CRC1=$1F
BBF @ 51622601    MATYPE=$CE01 [TS    MIS ACM ISSYI=1 NPD=1 RO=2 ISI=$01] UPL=$0D51 DFL=$E4F0 SYNC=$85 SYNCD=$40F8 CRC1=$64
BBF @ 51629937    MATYPE=$4200 [GS-C  MIS ACM ISSYI=0 GLT=0 RO=2 ISI=$00] UPL=$0000 DFL=$2938 SYNC=$20 SYNCD=$0000 CRC1=$D0
BBF @ 51631266    MATYPE=$4200 [GS-C  MIS ACM ISSYI=0 GLT=0 RO=2 ISI=$00] UPL=$0000 DFL=$2938 SYNC=$21 SYNCD=$0000 CRC1=$53



Mips I was trying to figure out were the MATYPE 9744 and CE01 came from if you get a chance can you look at that again? I looked at it and again and I get the same log file but looking at it with a hex editor it looks different than what your app shows it looks like there's some 42's it missed.

I found this one too "MATYPE=$CE01" am I wrong? but the log doesn't look right.
Thanks

For MATYPE=$9744, I use the DFL of the previous BB header to locate the following BB header. The previous BB header was at offset 51618856. So we have 51618856+10+($2938 / 8) = 51620185, which is the offset of the following BB header.

Similarly, for MATYPE=$CE01, we have 51620185+10+($4B34 / 8) = offset 51622601.

The next BB header will be at 51622601+10+($E4F0 / 8) = offset 51629937, which coincidentally gets back to a real BB header.

I agree that we're missing real BB headers. The one at offset 51620384 (highlighted in red in your screen grab) is possibly a real one. I'm saying "possibly" because I haven't validated its CRC, but the SYNC value of $18 is close to the SYNC value ($20) of the next valid BB header in the log file. In fact, if you keep searching for 420000 in hex, you'll find other possible BB headers with SYNC values from $19 to $1F, which occur right before the BB header at offset 51629937 (the sync values are continuous from $18 to $20, so these are likely the real BB headers).

The reason why my tool misses those BB headers is because of the DFL value of the BB header at offset 51618856. That value gets used to calculate the (wrong) offset of the next BB header, but since the CRC is valid, there's no reason to doubt the DFL value is corrupted.

One way the DFL value can be valid yet it takes us to a fake BB header is if data is missing. That is what I believe occurred here. You'll notice that the SYNC values of preceding BB headers incremented nicely up to $B2, and those of the following BB headers start incrementing nicely from $20. As I mentioned above, the SYNC values $18 to $1F are also present in the hex editor, so it looks like we're missing all the BB headers with SYNC values from $B3 to $17.

If stream data weren't missing, the DFL value would have likely led us to the next valid BB header (with SYNC value $B3). But since data is missing, the DFL value takes us in the middle of nowhere and you get MATYPE=$9744 (one of the fake BB headers).

What I find interesting is the fact that the CRC of the fake BB headers is valid. If the headers are fake, I wouldn't expect their CRC to be valid. Sure, it could happen randomly (1 out of 256 times), but every single fake BB header in the file has a valid CRC! It's unlikely that it would be transmitted like that, therefore the CRC must be getting recalculated/overwritten at reception (e.g. by the hardware, firmware, driver or app used to record the stream). That interferes with our ability to detect real BB headers from fake ones, and I don't see why it is being done.

If the CRC of fake BB headers were invalid, it would be possible to ignore those headers and look for the next real ones, but since the CRC is valid, there's no reason to ignore them and you end up with a series of fake BB headers in the log file.

To summarize, there are two problems here: data is missing from the stream, and the CRC of fake BB headers is being calculated/overwritten at reception.

EDIT: Updated text for clarity
andyinyakima

Avatar / Picture

Registered:
Posts: 1,520
Reply with quote  #33 
Keeping it simple, take a look at 12174 V 4172 at 123 W .

I have attached my header decode.

 
Attached Files
txt Untitled_Document_3.txt (764.19 KB, 18 views)


__________________
andyinyakima
-----------------------------------
Recs: TBS6983, TBS5925 __ Ant: 90cm on SG6100__OSes: Linux with v4l-updatelee drivers __ RPi2 with v4l-updatelee drivers

Open Book, Open Source, You have to Open it to Know what's in it!

midwestmac

Registered:
Posts: 2,275
Reply with quote  #34 
Thanks Mips I understand now I didn't check the DFL from to see if they all line up. Somewhere I read about a commercial reciever discarding packets but I've read so many different articles on this I can't remember maybe they were talking about GSE headers being discarded. More study.. Thanks Mips.

Thanks Andy got a keep it simple I guess. I'll look at some different ones.
For the heck of it here's the ones I've been playing with on KU. I've caught video on all of them at sometimes of the day.
AMC 15 12107 V 7141 fec (switching)
G18 12122 V 10796   (switching)
G16 12164 V 5151    (switching)
Satmex6@113w 12059 V 7933 (switching)... I haven't look at this one in a while. I saw ECTV EcuadorTV on it once

This one "below" shows encrypted with everything I look at it with Tsreader, transedit but its constantly pumping out video for someone
I guess what tsreader sees isn't wrong?
G17 12080 H 30000 fec (switching)
G17 12000 H 30000 fec (switching).. Can't lock this one very well

Thanks

__________________
Azbox Ultra, Pansat 2500, Prof7301,Tbs 6925,5980, Genpix 8psk card, Dektec 2137c, Hauppauge 950q
mips

Registered:
Posts: 529
Reply with quote  #35 
Quote:
Originally Posted by andyinyakima
Keeping it simple, take a look at 12174 V 4172 at 123 W .

I have attached my header decode.

The MATYPE field looks good. There seems to be a look of discontinuities in the SYNC values though. It could be normal for that TP, or perhaps a lot of BB headers are being skipped, perhaps as a result of some config options.

Are you locating the BB headers by searching for the hex pattern 42 00 00 00 or are you instead computing the location of the next BB header using the DFL field of the previous one?

mips

Registered:
Posts: 529
Reply with quote  #36 
Quote:
Originally Posted by midwestmac
Somewhere I read about a commercial reciever discarding packets but I've read so many different articles on this I can't remember maybe they were talking about GSE headers being discarded.

If GSE packets are getting discarded, then it doesn't seem to be implemented correctly.

I'm hoping there is a way to record all the GSE streams at once (without any filtering/discarding). If that can be achieved, then it should be possible to demux all the streams, a bit like IPCleaner does with IP feeds.
midwestmac

Registered:
Posts: 2,275
Reply with quote  #37 

Ok Mips uploaded a better recording I think?  The DFL seems to line up better I just checked three but the log I get from your program looks a lot better. I haven't gone into checking the CRC.
With crazycats streamreader demo in the past I had the inversion on auto. So taking a look at my signal info with crazyscan
it was inverted so I changed that to "ON" instead of auto. 
Then also in my streamreader.ini  I put the ";" in front of the "FrameMode=1" to disable it.

I'm experimenting. I'll upload another for you with inverted "on" and also enable the  "FrameMode=1" by removing the ";"
and see what you think of the two? 


__________________
Azbox Ultra, Pansat 2500, Prof7301,Tbs 6925,5980, Genpix 8psk card, Dektec 2137c, Hauppauge 950q
wejones

Avatar / Picture

Registered:
Posts: 5,640
Reply with quote  #38 
Hi
  I've been reading this thread, and the previous thread under the ACM/VCM topic, but I'm having a hard time understanding exactly what you're looking for and why.  In particular, it seems like you're looking for 42 00 00 .........  type sequences, but in reading through several of the links to documents on the subject, I don't see that sequence mentioned.  Also, in one of the documents that described the GSE headers, the number of bits/bytes didn't seem to correspond to the sequences posted here, and similar sequences I've seen.
  Also, I never saw any description of what a BB header should look like, only the GSE headers.  So I was wondering if someone could provide a quick explanation of what you're looking for here. 

  Another thing that I'm confused by is just what you're looking for in a transponder to be likely to have these BB/GSE headers.  My conclusion is that it's transponders that show up as "continuous" in Crazyscan, but I wasn't sure.  If so, will all continuous transponders be of this BB/GSE structure? 
 

  Anyway, yesterday, I came across one of those ACM/VCM signals that was switching between QPSK, 8PSK, 16APSK and 32APSK, so I recorded a bit of it.  The signal was showing up as being error free on Crazyscan.  When I looked at it with a binary editor, there were lots of those 42 00 00 ...... type sequences, but since I didn't know if this was what you guys were looking for, or if it is just a sequence that is often sent along with what you're looking for. 

   But this particular transponder, which I'm sure you guys must have looked at,  seemed to have something like 9 or 10 (+/- a couple) identical 26 byte sequences, followed by one more similar but different sequence just before a bunch of data is transmitted. Also, there are numerous single examples of similar sequences inside the data.

I show 3 examples of the multiple identical sequences followed by the start of data sequencess below.
I'm curious if these have anything to do with the BB/GSE things you guys are looking for, or are they something else.




42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 1B 60 A6 00 00 CF C1 59 00 02 00 80 12 C0 00 24 AB 01 C2 45 40 01



42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 15 B0 78 00 00 8B C1 59 00 02 00 80 12 C0 00 26 18 01 C2 45 40 01



42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 00 80 00 00 00 F7 E0 0C 00 06 00 00 00 00 00 00 00 00 F2 14 03 57
42 00 00 00 1F 70 44 00 00 8B C2 1D 00 02 00 80 43 C0 00 21 E9 01 D2 45 00 02
midwestmac

Registered:
Posts: 2,275
Reply with quote  #39 
Hey wejones, what you posted looks like a bbframe. I'm trying to find a good error free recording of one of these signals that are switching.
Its been hard to do. I'm using crazycats stream reader demo I have also tried the TBStsrecorder with the "GS" selected in the "Output Stream"

The BBheader will be 10 bytes like this one you have above "42 00 00 00 1F 70 44 00 00 8B" but the problem I'm having is the DFL (Data Field Length) were yours is "1F 70" which is 8048 (decimal) 8048 divided by 8 = 1006 bytes. Which is how far until the next "42 00"
Yours looks good on that part it divides by 8. the recordings I get some times the DFL won't divide by 8 
Let me post a diagram.. hold on sec

Here's how we get "42"  in (bits)

MAtype1-03-15_0320.png 
Then here's the BBheader broken down
bbheader2014-04-03_1937.png 





__________________
Azbox Ultra, Pansat 2500, Prof7301,Tbs 6925,5980, Genpix 8psk card, Dektec 2137c, Hauppauge 950q
wejones

Avatar / Picture

Registered:
Posts: 5,640
Reply with quote  #40 
OK, thanks. That helps a lot.  So in the repeating sequences, the "80"=128/8=16, which works out perfectly.

OK, then, in case these recordings might be useful, I can upload them tomorrow morning (they're kind of big, and I get almost free bandwidth if I upload before 8AM).   This transponder is on G28 at 4101 H 14213 . The content seems to be like internet traffic, like web pages and calls to Windows Update, etc, etc.   One ?web page? seemed to be related to air line traveling. 

Anyway thanks for the explanation.  I'll check to see if that DFL is consistently accurate. 


EDIT1:  BTW, I was receiving the transponder with a S/N of about 16.9 dB yesterday.  Today it's down a bit at 16.2.  Kind of borderline for when it's in 32 APSK, but it showed up as error free, but without TSREADER decoding, I couldn't tell if there were continuity errors from lost data.

EDIT2:  Also,  I made 2 recordings, one via the demo program, and one by just sending the Crazyscan output that normally goes to TSREADER to a file.  Both seemed to be pretty much the same as far as I could tell, but I wasn't sure what to look for.


midwestmac

Registered:
Posts: 2,275
Reply with quote  #41 
The GSE header is right after the 10 bytes.  Most of the GSE headers I've seen are 13 bytes but that can change with the last part of the GSE header there's a "label" which can be either  3 or 6 bytes. Most of the ones I have had a 3 byte label. I think? And I believe from what I read also that some could have no label.
But I haven't  got into all that part yet.

That might be a recording. What did you record that one with?

Added: About the GSEheader  Look at Andys post #7 in this thread he breaks down the GSE.

Thanks I just saw you post above.

ADDED:
I copied this from this "oatao" website. If you want to see how much more it gets complicated[smile]

""The GSE Header is composed of the fields
shown in Figure 2. The unshaded fields are always present,
while the shaded fields may be omitted depending on the
preceding control fields in the first 4 bits of the GSE Header.
The presence of possible Extension Headers is determined by
the Protocol Type value, following ULE-like rules and IANA
allocations. The minimum GSE Header length is therefore 2
Bytes.""


2014-04-24_1202.png 


__________________
Azbox Ultra, Pansat 2500, Prof7301,Tbs 6925,5980, Genpix 8psk card, Dektec 2137c, Hauppauge 950q
midwestmac

Registered:
Posts: 2,275
Reply with quote  #42 
Oh by the way, Back to the BBheader in my recordings I have a "42 00 00 00" and a "42 00 02 00". Have to see what mips thinks the "42 00 02 00" I get a DFL that doesn't line up right so I don't know? I'll have to look at it more.
But the 42 00 00 00 the DFL lines up fine so far.

__________________
Azbox Ultra, Pansat 2500, Prof7301,Tbs 6925,5980, Genpix 8psk card, Dektec 2137c, Hauppauge 950q
mips

Registered:
Posts: 529
Reply with quote  #43 
Quote:
Originally Posted by midwestmac

Ok Mips uploaded a better recording I think?  The DFL seems to line up better I just checked three but the log I get from your program looks a lot better. I haven't gone into checking the CRC.



Thanks again for the files.

BTW you can quickly check for invalid CRCs in the log file by searching for "CRC0". If you get too many of those, the recording is likely in bad shape. It acts a bit like the continuity counter in TS streams in that it helps you detect problems in the stream.

If you get none, then it means they're all "CRC1"s and are valid. That's a good start. There could still be fake BB headers with a valid CRC but it's a first step to weed out corrupted recordings.

mips

Registered:
Posts: 529
Reply with quote  #44 
Quote:
Originally Posted by wejones

  Another thing that I'm confused by is just what you're looking for in a transponder to be likely to have these BB/GSE headers.  My conclusion is that it's transponders that show up as "continuous" in Crazyscan, but I wasn't sure.  If so, will all continuous transponders be of this BB/GSE structure? 
 

  Anyway, yesterday, I came across one of those ACM/VCM signals that was switching between QPSK, 8PSK, 16APSK and 32APSK, so I recorded a bit of it.  The signal was showing up as being error free on Crazyscan.  When I looked at it with a binary editor, there were lots of those 42 00 00 ...... type sequences, but since I didn't know if this was what you guys were looking for, or if it is just a sequence that is often sent along with what you're looking for. 


The BB headers can be GSE continuous (GS-C), GSE packetized or TS. I've only seen GS-C recordings so far, but my guess is BB headers in TS mode are not the same as regular transport streams. I'm not sure how all those stream types would be reported in Crazyscan though.

When data is missing from recordings, the fake headers often report non-sensical values e.g. in a GS-C stream, you also see "GS-P", "GS-HE" or "TS" in the log file. Those headers could be legit though (I don't recall reading anything in the spec that says you can't mix stream types), so I usually look for those strings in the log file, then look for inconsistencies near their location, such as values that are not divisible by 8, UPL fields that are non-zero when they should, etc.

The hex sequence 42 00 00 works with GS-C streams but other stream types will have different sequences (because of the bit pattern for MATYPE1), so only searching for 42 00 00 in a hex editor will make you skip over legitimate BB headers if other stream types are also present. My tool should be able to interpret any stream type, even mixed ones.
mips

Registered:
Posts: 529
Reply with quote  #45 
Quote:
Originally Posted by wejones

OK, then, in case these recordings might be useful, I can upload them tomorrow morning (they're kind of big, and I get almost free bandwidth if I upload before 8AM).

That would be great, thanks. To save your bandwidth and avoid uploading corrupted files, you can look for invalid CRCs ("CRC0") in the log file as I mentioned in a previous post, and try searching for non-sensical values (e.g. BB headers with an unexpected stream type).

Previous Topic | Next Topic
Print
Reply

Quick Navigation:

Easily create a Forum Website with Website Toolbox.

Fellow Members, your posts are welcome here! Do not worry about posting everything perfect. Different receivers and LNB's will give you different Frequencies and Symbol Rates. Some set top boxes, PCI cards and USB receivers, Do Not Require all of the same information that others may need. It is not Required to post everything that others may need to tune in a feed. It is just most important to share the find. We can always adjust the Frequency and Symbol Rates and try the various Modulations and FEC's on our own receivers until we get a lock and then give a polite reply with what works for your receiver, as that information might help others as well. We all appreciate the efforts and energy of the Posters!

Thank You for Visiting GOFASTMOTORSPORTS.com - Keep Your Eyes on the Sky and the Track!