Welcome to GOFASTMOTORSPORTS.com ... ... ... Green Flag - Green Flag - Go Go Go ... ... Brought to you by Ricks Satellite - Where the Big Dish Rules!

Ricks Satellite Wildfeed and Backhaul Forum
Register Latest Topics
 
 
 


Reply
  Author   Comment   Page 2 of 9      Prev   1   2   3   4   5   Next   »
bluzee

Registered:
Posts: 2,186
Reply with quote  #16 
Heya mips!  I like your app.  Thanks for that.

No worries merkin,  I see what you mean now.  Tried it on the 4140H stream you mentioned and I can see how that works.  Definitely two different shows coming down the pipe at the same time. I remember looking at that before, but had no idea there was more than one file being streamed. 

Yes, the issue with null data blocking IPcleaner is what i was referring to.  Instead of using a hex editor  I just used tail -c to move past that spot in the TS dump.  Now that I think of it I could have been redirecting that output to a fifo as well and then have IPcleaner process from the fifo instead of rewriting all that data.  If I plug away at this stuff long enough I eventually get it figured.

Home sick with the plague today, so just playing with this to keep busy until there is hockey.
mips

Registered:
Posts: 529
Reply with quote  #17 
Quote:
Originally Posted by bluzee
Heya mips!  I like your app.  Thanks for that.

No worries merkin,  I see what you mean now.  Tried it on the 4140H stream you mentioned and I can see how that works.  Definitely two different shows coming down the pipe at the same time. I remember looking at that before, but had no idea there was more than one file being streamed. 

Yes, the issue with null data blocking IPcleaner is what i was referring to.  Instead of using a hex editor  I just used tail -c to move past that spot in the TS dump.  Now that I think of it I could have been redirecting that output to a fifo as well and then have IPcleaner process from the fifo instead of rewriting all that data.  If I plug away at this stuff long enough I eventually get it figured.


You're welcome

As merkin mentioned, the "sub" argument should help to separate the streams. See the readme for an example showing how to extract the first 3 substreams from a PitchBlue feed.

Unless there's something unusual going on, IPCleaner probably doesn't stop at the null data, it should continue all the way to the end of the file/stream. What's likely happening is that it's extracting a specific substream (the default value is 1) and that substream is not present after the null data, but the rest of the file is still getting "processed" (skipped really).

If you use "-sub=2", then the first substream in the file will get skipped and it will start extracting the 2nd substream in the file (probably after the null data). And so on.

bluzee

Registered:
Posts: 2,186
Reply with quote  #18 
What I've noticed that makes me say that is sometimes I'll leave it running to record a show and when I get back everything is still running, but no data is being written to the cleaned mpg file  null or otherwise.  If I kill it off and let it restart it will start writing data again.  It may just be null data or it may actually be video again.  It's recorded until the first stream disappeared I guess.  Sometimes it will exit out right before the new file starts to stream.  That is ideal as then the script will just loop and start over again.

I'll play with it using the new -sub switch and see if that is where the data is actually located at the points where it stops writing data to the first mpg file.

It's certainly interesting stuff to play around with.
mips

Registered:
Posts: 529
Reply with quote  #19 
Quote:
Originally Posted by bluzee
What I've noticed that makes me say that is sometimes I'll leave it running to record a show and when I get back everything is still running, but no data is being written to the cleaned mpg file  null or otherwise.  If I kill it off and let it restart it will start writing data again.  It may just be null data or it may actually be video again. 

Your description is consistent with the presence of multiple substreams.

Let's say there are three substreams being transmitted. You start recording the first substream and get back while the 3rd substream is being transmitted. At that point, you'll notice that no more data is being written to the cleaned mpg file (since only the first substream was being extracted). If you now kill everything and restart IPCleaner, data will get written again to a new cleaned mpg file because the 3rd substream will now be considered the first one in the stream. In both runs, the first substream was being extracted ("-sub=1" implied or explicitly stated).

IPCleaner internally labels each substream based on the order in which it appears in the stream. Whenever you restart IPCleaner, the substream number starts over at 1.


merkin

Registered:
Posts: 741
Reply with quote  #20 
But mips,
I had a file that gave me the same issues (just one to date).  IPcleaner would not process it until I deleted a large amount of bytes at the start of the file (just until I recognized another subid). After manual hex editing ipcleaner worked as usual. I will dig around for it, and pass it along if interested.  Just not sure its on the HDD still.

Can't say it enough, but thanks again for sharing the tool.
mips

Registered:
Posts: 529
Reply with quote  #21 
Quote:
Originally Posted by merkin
But mips,
I had a file that gave me the same issues (just one to date).  IPcleaner would not process it until I deleted a large amount of bytes at the start of the file (just until I recognized another subid). After manual hex editing ipcleaner worked as usual. I will dig around for it, and pass it along if interested.  Just not sure its on the HDD still.


If you still have that file, I sure would like to take a look at it. bluzee and you might be right, but if that's the case, it sounds like something needs to be fixed in IPCleaner. Personally, I've never encountered such a case.

bluzee

Registered:
Posts: 2,186
Reply with quote  #22 
OK, if I understand correctly I start IPcleaner on a pid with -sub=1,  there is a file being streamed, it cleans that and writes it to the mpg file I specified.  That file stream now ends and a new file starts streaming.  The new stream is now actually -sub=2 because I have not restarted IPcleaner and it has identified it as a second substream.  Since IPcleaner was started with -sub=1 it stops writing data because this second file stream isn't -sub=1 anymore it's -sub=2. 

If I start multiple instances of IPcleaner then, one -sub=1, one -sub=2 etc. that should keep the process going then?  The second instance will ignore the first file download but grab the second and so on? All writing to a different file name of course. 

If I dump the TS to a file instead of reading a FIFO would I then run  IPcleaner several times on the TS dump using an incremental -sub=#  each time? 
mips

Registered:
Posts: 529
Reply with quote  #23 
Quote:
Originally Posted by bluzee
OK, if I understand correctly I start IPcleaner on a pid with -sub=1,  there is a file being streamed, it cleans that and writes it to the mpg file I specified.  That file stream now ends and a new file starts streaming.  The new stream is now actually -sub=2 because I have not restarted IPcleaner and it has identified it as a second substream.  Since IPcleaner was started with -sub=1 it stops writing data because this second file stream isn't -sub=1 anymore it's -sub=2. 

You got it.

Quote:
Originally Posted by bluzee

If I start multiple instances of IPcleaner then, one -sub=1, one -sub=2 etc. that should keep the process going then?  The second instance will ignore the first file download but grab the second and so on? All writing to a different file name of course. 

Yes it will, as long as each IPCleaner instance is processing the exact same stream byte-per-byte, otherwise the substream numbers may not relate from one instance to another.

Quote:
Originally Posted by bluzee

If I dump the TS to a file instead of reading a FIFO would I then run  IPcleaner several times on the TS dump using an incremental -sub=#  each time? 

Yes.

I've uploaded v0.38. This version shows how many substreams are present in a file using "-subinfo". That comes in handy to know when to stop incrementing the "sub=#" value.

EDIT: Download link removed. See http://rickcaylor.websitetoolbox.com/post/IPCleaner-software-Only-Post-File-Updates-Here-5924836 for latest version
bluzee

Registered:
Posts: 2,186
Reply with quote  #24 
Quote:
I've uploaded v0.38 http://rickcaylor.websitetoolbox.com/post/IPCleaner-software-Only-Post-File-Updates-Here-5924836. This version shows how many substreams are present in a file using "-subinfo". That comes in handy to know when to stop incrementing the "sub=#" value.


Awesome.

OK,  that info is most helpful.   Reading the man page for tee...  Alrighty then, wait and see if anything comes on after Deepak Chopra.

Edit:  Works exactly as advertized.    The trick for realtime(ish) cleaning appears to be tee.  Make yourself a whole wack of pipes, then instead of redirecting the TS stream to just one fifo pipe it to tee and have tee write the stream to all the pipes.  Once that is going then fire up IPcleaner on all the pipes using incremental -sub=#.  It'll start writing files skipping to the next pipe when ever a new file starts to stream.  Stops when the files stop streaming or you run out of pipes.

The command to pipe to tee is like this......

$dvbstream [tuning switches] -o 8192 | tee > pipe0 pipe1 pipe2 pipe3 etc.

I wrote a script that starts an IPcleaner instance for every pipe in the directory. Can share if anyone is interested.  I think I have it right this time.....


okidokios

Registered:
Posts: 700
Reply with quote  #25 
hello i tried ipcleaner with some pathfire feeds but when i record the full mux and when i do the ipcleaner the cap of 4gb just give one a 50mb file wich is just a comercial and part of the begining of another commercial is there any way to get all files that were sent in the mux? without extracting one by one with the sub command?
bluzee

Registered:
Posts: 2,186
Reply with quote  #26 
I think you'll have to run with the -sub= multiple times for multiple files.  Run through the first time with -subinfo also.  When it's done it'll spit out a list of all the available substreams in your TS dump.  You can then figure out based on the packet numbers which ones are big enough streams to be shows.  You can start up multiple instances of IPcleaner to get the remaining substreams out.   How many will depend on how fast your computer is I suppose.

Don't know what you mean by 4gb cap.  IPcleaner doesn't have a 4gb cap. Unless you are running some  ancient file system....
mips

Registered:
Posts: 529
Reply with quote  #27 
The ability to extract multiple streams automatically without having to specify the -sub parameter has not been a high priority since a workaround is available. You can use the 'for' command at the command prompt:
ipcleaner <input_file> nul -pid=N -subinfo
for /L %a in (1,1,M) do ipcleaner <input_file> sub%a.ts -pid=N -sub=%a

Replace N by your pid value and M by the highest substream to extract. The first line will output the substream info for pid N, which you can use to determine M. The second line extracts all the substreams from 1 to M.

okidokios

Registered:
Posts: 700
Reply with quote  #28 
I mean a 4gb caped (recorded) file from pathfire, well i tested and extract all files from the recorded mux, i used 010 editor wich give me all files in one the good thing with this tool is that pathfire sometimes send sd and hd files at the same time with 010 only get the hd files and skip the sd or viceverse.

by the way i notice too is that the split files sometimes at the end of the video have part of other video but glitchy is weird, this is a great tool it took some time to get files but is very nice tool, will be great if somebody can make a gui version or a plugin for tsreader and get files while they sent it

Quote:
Originally Posted by bluzee
I think you'll have to run with the -sub= multiple times for multiple files.  Run through the first time with -subinfo also.  When it's done it'll spit out a list of all the available substreams in your TS dump.  You can then figure out based on the packet numbers which ones are big enough streams to be shows.  You can start up multiple instances of IPcleaner to get the remaining substreams out.   How many will depend on how fast your computer is I suppose.

Don't know what you mean by 4gb cap.  IPcleaner doesn't have a 4gb cap. Unless you are running some  ancient file system....
bluzee

Registered:
Posts: 2,186
Reply with quote  #29 
Quote:
and get files while they sent it


If you are familiar with Linux have a look at the thread titled IP Cleaning.  You can get files while they send them if you can use Linux.

cap = capture...   Sorry, I'm not always the quickest off the draw.
Omkar

Registered:
Posts: 24
Reply with quote  #30 
Quote:
Originally Posted by mips
Quote:
Originally Posted by bluzee
OK, if I understand correctly I start IPcleaner on a pid with -sub=1,  there is a file being streamed, it cleans that and writes it to the mpg file I specified.  That file stream now ends and a new file starts streaming.  The new stream is now actually -sub=2 because I have not restarted IPcleaner and it has identified it as a second substream.  Since IPcleaner was started with -sub=1 it stops writing data because this second file stream isn't -sub=1 anymore it's -sub=2. 
<br>You got it.<br><br>
Quote:
Originally Posted by bluzee
<br>If I start multiple instances of IPcleaner then, one -sub=1, one -sub=2 etc. that should keep the process going then?  The second instance will ignore the first file download but grab the second and so on? All writing to a different file name of course. 
<br>Yes it will, as long as each IPCleaner instance is processing the exact same stream byte-per-byte, otherwise the substream numbers may not relate from one instance to another.<br><br>
Quote:
Originally Posted by bluzee
<br>If I dump the TS to a file instead of reading a FIFO would I then run  IPcleaner several times on the TS dump using an incremental -sub=#  each time?  <br>
<br>Yes.<br><br>I've uploaded v0.38 (http://www.etymonix.com/upload/IPcleaner%20v0.38beta.zip). This version shows how many substreams are present in a file using "-subinfo". That comes in handy to know when to stop incrementing the "sub=#" value.<br>


It gives file not found error. Google also not showing anything about IPcleaner software.
Previous Topic | Next Topic
Print
Reply

Quick Navigation:

Fellow Members, your posts are welcome here! Do not worry about posting everything perfect. Different receivers and LNB's will give you different Frequencies and Symbol Rates. Some set top boxes, PCI cards and USB receivers, Do Not Require all of the same information that others may need. It is not Required to post everything that others may need to tune in a feed. It is just most important to share the find. We can always adjust the Frequency and Symbol Rates and try the various Modulations and FEC's on our own receivers until we get a lock and then give a polite reply with what works for your receiver, as that information might help others as well. We all appreciate the efforts and energy of the Posters!

Thank You for Visiting GOFASTMOTORSPORTS.com - Keep Your Eyes on the Sky and the Track!