Welcome to GOFASTMOTORSPORTS.com ... ... ... Green Flag - Green Flag - Go Go Go ... ... Brought to you by Ricks Satellite - Where the Big Dish Rules!

Ricks Satellite Wildfeed and Backhaul Forum
Register Latest Topics
 
 
 


Reply
  Author   Comment   Page 3 of 3      Prev   1   2   3
wejones

Avatar / Picture

Registered:
Posts: 5,641
Reply with quote  #31 
Well, I got up early, and downloaded that 5 GB file, and 2 other files during the off hours period for my wild blue ISP, which has a separate bandwidth (I hope).
After downloading it, I went back and started reading some of the WIKI pages you posted about. {I know...I should have done that first.}    Anyway, I think I mis-understood what these files were supposed to be doing.  I was under the impression that this was a completely Windows based implementation from sat signal to graphics.  But now, if I'm reading this right, it's supposed to be two computers, one, a linux computer running the server (which I assume is in the 5 GB file), and two, a Windows computer running the client that takes data from the linux computer and processes the files into graphics.
   If I'm understanding correctly, what you have done so far is only the Windows side, which probably didn't even use the 5 GB file.  And if you can find an active NOAAPORT server on the web, this program should be able to spit out graphics.  I'm guessing that the url you connected to perhaps wasn't an active server, but instead a server that has some of the boilerplate graphics, like the basic maps that the real time images are superimposed upon????   Anyway, just guessing here. 
      If what I think now is correct, then I'm thinking that perhaps I should be using an older 32-bit computer (it could have been a 64 bit computer, but I set it up as 32 bit) which runs XP windows, but has a VMWARE installation of Ubuntu on it.  Problem is, however that right now, the only sat cards I have for that computer that would be accessable to the Ubuntu would be a TeVII and a PROF-7500, neither of which can do VCM. I could move my 6925 back to that computer, but VMWARE only has access to the USB receivers,  not to cards. It would be the same if I installed the server on an RPi, ie only access to USB receivers.  
    I also have an external hard disk with a very old Ubuntu, that I can boot to instead of XP, so that it could access the 6925, however I have a lot of important (to me) things running on that computer that would have to be moved somewhere. 

    Anyway, I'm hoping I'm wrong, and the server can be run from windows, otherwise, until I can come across another computer to put Linux on, or buy a 5925 USB receiver to use with the VMWARE Ubuntu, my best bet would be to figure out how to capture individual files, and somehow feed them to the windows graphics program, bypassing the server.  And I think that is what you're trying to do now???

EDIT:  BTW, I still have wireshark up from yesterday, with the info I captured from PID-102. I THOUGHT that wireshark could save this file to a raw file equivalent to the original stream, but I can't get that to work.  I can save it to a variety of stream analyzer program formats, but they have the original binary inside a bunch of headers, rather that just the raw stream.
   There is an option to export selected packet bytes, which is supposed to be the raw stream, but this is grayed out, and there doesn't seem to be any way to select packets for it to export.
   Just curious re whether you know how to make wireshark export the raw stream rather than in non-raw formats???


midwestmac

Registered:
Posts: 2,275
Reply with quote  #32 
Sounds like you got it! Yes I thought these files would be a way  to connect to satellite too. I'll have to go inside and look at the file more closely.
I think the 5 GB is the java part of the program that allows a user to interface with weather maps..edit..add text to the image etc...  Then send that weather map back into the network via the server ie port 9581.
I read some info on a pdf that leads me to believe  that all the weather data is stored on a central data computer  compressed in binary to save space. When a client requests a radar map a marked or tagged for example..  <radar>(.xml file) the request is sent to the server.. the server decompresses the data(.xml file) and gives the radar map to the client. The client can edit the radar map then send it back on the network if this new (.xml file) with the right WMO header matches the "properly" tagged.xml file.
If not "tagged" properly the server will reject the file before it sends it back to  be stored. I think either the server? or a higher up computer? compresses the "binary" file and stores it properly.

Here's the very large pdf link that is almost a book. If you want to look at it and maybe give your conclusions. If you want?
http://www.unidata.ucar.edu/software/awips2/doc/System_Manual_13.4.1.pdf

re 32bit or 64bit I installed the cave client on win7 64bit seems to work fine,

re: bypass the server?  Yes, since then I was trying to install the netcdf on linux Ubuntu for now, I think redhat centos would be better or "RheL". The netcdf requires a software/library "hdf5"(Hierarchical Data Format) and also either zlib or szip or both. from what I can tell its best to use 64bit on a x86_64 OS because some of these software and libraries  just causes error building problems.
The VMware might work I haven't tried that.
Anyways my goal now was to install netcdf4, then from a terminal do (example) "ncdump <file>"
and I'm not sure if this will work yet.

re: wireshark? I'm not sure I looked on the net and couldn't find anything.
I did find a topic on netfcdf on microsoft Exel but, ONLY Exel 2007 can be used and I have 2010


__________________
Azbox Ultra, Pansat 2500, Prof7301,Tbs 6925,5980, Genpix 8psk card, Dektec 2137c, Hauppauge 950q
wejones

Avatar / Picture

Registered:
Posts: 5,641
Reply with quote  #33 
Quote:
Originally Posted by midwestmac
.
I did find a topic on netfcdf on microsoft Exel but, ONLY Exel 2007 can be used and I have 2010


Strange.  I can't imagine why Excel 2010 couldn't read a file written for 2007.  I have Excel 2007 on one of my computers here.  I could try it. 

I've been playing with the PID 102 all day here.  It's the one with GOES data.  It seems to idle most of the time, sending out a short packet about every 30 seconds.  Then, whenever an image becomes available, which I think is about every 10 or 15 minutes (I forget), it spits out the data. 
  Today I had my 6983 feeding data to wireshark, so I could see when the data was in idle mode, and on my 6925, I had TSREADER in IP/DVB mode.  TSREADER has 3 different modes to record in, (1) Payload only, (2) Payload and IP headers, and (3) Payload an IP headers and MPE Header.  
I don't remember what MPE headers were, and I thought I could identify where I was easier using the #2, so that's what I used.  Got a pretty good recording.  
   The wireshark output shows a few packets from my computer's network, and each packet contained an ethernet section that isn't in the direct TSREADER recording.  
   I displayed my recording with a hex editor, and found the beginning of the weather image transmission, and compared to the wireshark output.

noaaport-1127a.jpg 
     {Sorry the image is a bit blurry, did a remote screen copy from my UHD computer}
On the right, wireshark shows 3 idle packets, at 269,299,329 seconds, then apparently the beginning of the transmission.  The Internet Protocol (highlighted, User datagram, and DATA) are included in the hex editor view of the recording at left. Each packet starts with hex 45.  The hex output starts with the packet I highlighted at the right, and I've highlighted the start of the next packet on the left, near the end of the 4th line down.
   So I'm guessing that if I had recorded in the payload only mode, it might have eliminated the IP and datagram headers, but it may have been harder to locate the beginning of the file, but I'll try to work on that.
    But If I can reconstruct the file, perhaps it can be decompressed and displayed.


  I had mentioned earlier that I thought I had read about free NOAAPORT servers on the web.  Well I ran across a discussion at
https://stormtrack.org/community/threads/awips-2-availability.27702/
that says that this was once the case due to a federal grant, but that now there are very few if any free servers.  It mentioned one commercial server (at a company my wife used to work for) that offers the service to hobbiests for about $2000/month or $6000/month for commercial use.  Then there was a server that apparently was established by an individual, who offers it for $20/month.  Pretty big difference.  [smile]
    But the discussion also mentions putting the home system together using VMWARE, but I wasn't sure exactly what he was doing. 

But anyway, this PID 102 seems to spit out data infrequently enough that I guessing we could automate grabbing the images, unzipping them and displaying them, then wait for the next image, once we figure out how to reconstruct the file and decompress it.  Some of those other PIDs spit out data so fast that there isn't much idle time between the files to process them.

 
EDIT:  I think it's more complicated than I thought.  Ie it looks like there are multiple files sent between the idle periods, so we'd have to identify the type of file.  Also, the DATA portion of the packet seems to be more than just the file content.  Ie the first portion of the DATA seems to be relatively constant, along with counters. So it seems there are headers within the data.  I think I have the format for that somewhere. I tried to decode it once before, but didn't understand how to remove the other headers.  I still think it's doable, but complicated. So maybe another look at the NOAA programs, and the document you found.  


EDIT2:  I finally opened up that 5 GB tar file.  I had assumed that it was linux source, but I see that it is virtually all  .RPM files.  I didn't know what a RPM file was, but googled and see that it is a Red Hat installation format.  I used to have a Red Hat computer, back in the 1990s, but haven't touched it in 15 years.  The info I saw on it said that a program called alien will convert it into a format usable by other linux systems.  I still don't know what the package does, but I did see a netcdf file in there.  

BTW, part of the reason I got MULTIPLE files between the idle periods was that I had mistakenly left the thing running.  I thought that I had only recorded for a couple minutes, but instead had left wireshark recording for several hours, collecting nearly a GB of data.  No wonder that there were many files there.






 




midwestmac

Registered:
Posts: 2,275
Reply with quote  #34 
Nice job, I see in your pic the 90 bytes, but I can't tell what the next packet 300 bytes starts with?
Because, I see TIGQ43 KNES 271720.
Playing around with the Cave program. That's the WMO header which one would use to look for a specific data file.
I think? if I have this right.
The TIGQ43 tells about the contents of the file each letter would mean something from the NOAA alpha table
The KNES tells the international location 
The 271720 would be the AWIPS ID #

I haven't figured out the Alpha table yet.
BTW, I found a "raw" expression in the wireshark filters I don't if that would help any?

PS. From my post above, I am assuming the data is compressed because the netcdf4 required szip and zlib

ADDED: The AWips ID, is not top secret it helps to find the data your looking for and its supposed to match the
               the WMO header in some way. I found the start of that 300byt packet I think its? ¨A0 00 09 36¨ my recording has that.     
               And nevermind, these are compressed




__________________
Azbox Ultra, Pansat 2500, Prof7301,Tbs 6925,5980, Genpix 8psk card, Dektec 2137c, Hauppauge 950q
midwestmac

Registered:
Posts: 2,275
Reply with quote  #35 
I have taken a break then go back to my recording of this. I think the images might be jpeg-2000 compressed?
Looking at it with hexeditor I found Jasper which I read is jpeg-2000  but I can't seem to find the right place to strip away part of the file.
Or is this a Grib file with a jpeg-2000 inside.
I downloaded a jpeg2000 viewer but have had no luck yet saving then opening a ".jp2" file.
Also Ncfd4 files I think end in ".nc" so I don't know if having ncdf4 installed would help or not.


NOAAhexedit.png 



__________________
Azbox Ultra, Pansat 2500, Prof7301,Tbs 6925,5980, Genpix 8psk card, Dektec 2137c, Hauppauge 950q
weather01089

Registered:
Posts: 40
Reply with quote  #36 

I am a volunteer spotter coordinator for the weather service Skywarn program. I have been ingesting all kinds of weather data from multiple sources for decades.

The NOAAPORT signal is designed to feed NWS offices with the mass data they need. Text products such as watches, warnings, and surface, marine and aviation observations, radar, satellite, and forecast model data are on the stream and the PID's listed.

The headers are broken down like this, using the example above:

TIGQ43 KNES 271720
TIGQ43 is the product ID, tells you what it is.
KNEX is the issuing NWS point or office K for the first letter indicates the US
271720 is the DTG, Date Time group. In this case, the 27th, at 1720 GMT

The latest PID's were added to support the new GOES-R series of satellites. Currently test data from GOES16 is being sent on them.

You can refer to the list above for whats on the rest. You need the multiple PID's to get the entire feed.

WXMESG will ingest watches, forecasts, and warnings from the NOAAPORT feed, normally PID 101.


The NWS uses EDEX as ingest software, so you need an EDEX server as a frontend. EDEX uses LDM from unidata to interface with noaaport. LDM sends them to EDEX via a small program called Edexbridge. Edex decodes the NOAAPORT products and saves them in various formats including HDF5, Text and NETCDF among others. You then need a workstation running CAVE that connects to EDEX and views all the products. The EDEX/CAVE combination is part of the AWIPSII suite. I have a 12 ft dish and in the NE US it barely gets by with 12db CN. This is an example pic. Ingested by EDEX and viewed on CAVE from the GOES-16 satellite. MESOscale pictures like this are centered over severe weather and produced every minute.
stormsco-20170508_215827.jpg 

 


weather01089

Registered:
Posts: 40
Reply with quote  #37 
As a side note to my previous post, I am using a NOVRA S300-N receiver. I am also testing a TBS
-6903 card, which seems to work as well. Keep in mind the majority of this data sent is just that, Data, that has to be interpreted and overlaid on a map by CAVE.
merkin

Registered:
Posts: 770
Reply with quote  #38 
thanks for sharing.
it seems all this software is readily available so anyone could build.

what tuner/demod combo is in the s300n?

have you ever played around with other acm/vcm transponders with the s300n?
we have not had much success with acm/vcm and whether or not its a hardware or software issue is still unknown to me.
weather01089

Registered:
Posts: 40
Reply with quote  #39 
The s300 is a dedicated small receiver with an Ethernet port. I am able however to get the signal with the tbs6903 card, which supports vcm. Its the receiver used by most weather service offices and universities working with noaaport.

This is the website for it:
http://novra.com/product-line/s300-dvb-s2-ip-satellite-data-receiver/

The TBS6903 has VCM support, and I have it working fine as well.




midwestmac

Registered:
Posts: 2,275
Reply with quote  #40 
Thanks for sharing Weather01089. been a while since we were playing around with this. We were trying to capture the signal, find a data file, then trying to open the binary data  file with whatever app could read the most .
I can't get this signal since they changed it a few years ago but, wejones was able to stream a pid with tsreader over his network and look at the packets beeing sent.
pretty much so we don't have to have 2 computers to view a picture like you posted above.

So I guess you meant you have TBS 6903?

Does your TBS card work with EDEX server? and is that server on Redhat or centos?
I have this link for EDEX server  and cave is that the right?
http://www.unidata.ucar.edu/software/awips2/doc/install.html
.

__________________
Azbox Ultra, Pansat 2500, Prof7301,Tbs 6925,5980, Genpix 8psk card, Dektec 2137c, Hauppauge 950q
weather01089

Registered:
Posts: 40
Reply with quote  #41 
The TBS6903 does work with ldm/edex. I am not using that version that's on the unidata site, although I have tried it. Unidata wrote the ldm software the weather service uses that bridges NOAAPORT and edex with modules called noaaportIngester, and edexBridge.

The unidata version installs far easier than the weather service version. The NWS version has had a number of issues that had to be overcome to get it running. Not for the faint hearted.

But if you feel brave, you can take a shot at the NWS version. It is located  here:
https://collaborate2.nws.noaa.gov/partners/

Current version is 17.1.1

I am using Centos. Had to use 6.8, had issues with 6.9 and the 7.X versions.

Novra also just came out with another of their commercial receivers that has more throughput.

But the 6903 card does work, and thus far, pretty well.


midwestmac

Registered:
Posts: 2,275
Reply with quote  #42 
Ok thanks Ray, nice to get input from someone who has done this. I'll have to use my brother in laws dish to get this signal again but, this might help others who want to try it
I have TBS 6925 card which should work I think? 
Thanks!!

__________________
Azbox Ultra, Pansat 2500, Prof7301,Tbs 6925,5980, Genpix 8psk card, Dektec 2137c, Hauppauge 950q
weather01089

Registered:
Posts: 40
Reply with quote  #43 
On paper looks like the 6925 should work. The 6903  is what I have, I misposted the model. Ill correct that LOL The 6903 has 2 ports only.
Previous Topic | Next Topic
Print
Reply

Quick Navigation:

Easily create a Forum Website with Website Toolbox.

Fellow Members, your posts are welcome here! Do not worry about posting everything perfect. Different receivers and LNB's will give you different Frequencies and Symbol Rates. Some set top boxes, PCI cards and USB receivers, Do Not Require all of the same information that others may need. It is not Required to post everything that others may need to tune in a feed. It is just most important to share the find. We can always adjust the Frequency and Symbol Rates and try the various Modulations and FEC's on our own receivers until we get a lock and then give a polite reply with what works for your receiver, as that information might help others as well. We all appreciate the efforts and energy of the Posters!

Thank You for Visiting GOFASTMOTORSPORTS.com - Keep Your Eyes on the Sky and the Track!