Add simh image file support to tape device #100
Comments
uweseimet
added
enhancement
|
@Pacjunk The issue_100 branch now contains initial code that for .tap files supports the simh file format. I have already tested this code with my limited setup. tar can successfully write/read to/from .tap files in the new format. This means that at least consecutive data blocks are dealt with correctly, just like before with the proprietary raw format. When you start testing (e.g. by labelling a blank "tape") I expect some issues, new ones or familiar ones, we will see ... |
|
Wow, you have been busy overnight! I'll see if I can make some time... |
|
OK, I rebuilt on the issue_100 branch and did some more testing (sorry still haven't connected that tape drive yet)... I initialised (labelled) a tape (success) then I mounted that tape using the label (success) then I did a "dir" on the blank tape (success) then I attempted to copy a file to the tape (failed parity errors, device errors) Another "dir" on the tape (more parity errors etc) I then connected a pre-existing simh image from bitsavers. I mounted the tape (success!) Then a "dir" on the tape to check contents (parity errors, but did display the first file name on the tape - not all) That is all I had time for tonight. I'll try to get to the physical tape drive soon... |
|
@Pacjunk I'm afraid I don't know what a parity error in this context means. The tape code does not deal with parity. My guess is that your driver is not satisfied with the data it sees or there are (still) issues with the tape-related data returned by REQUEST SENSE. But the data returned appear to be better than before now, because on first sight I could not find any attempt to reverse-space anymore in the logs. I just fixed some issues and improved the logging, please use the latest avallable issue_100 branch for your next test. There is now also initial reverse-spacing support. Reminder: Always add information on the commit ID you were testing. Please also do not forget to verify your web UI issue (see my comment in the issue_93 ticket about the web UI). Regarding init.txt there is no command writing the label. Line 105 appears to successfully read it, so it must have been created before. Please ensure that no write operations are missing in the logs. I noticed that the existing simh files contain private data records, which SCSI2Pi will not be able to process. See the simh file specification for details on private records. This is not necessarily an issue, but it means that there are custom data in the simh files that only the software that created these files can make use of. Or the detection of this type is still a bug in SCSI2Pi. And one more thing: As far as I can tell you are testing with an optimized build. If this is the case, please test with a debug build, so that assertions are enabled. |
|
Looks like my previous logs were truncated... I ran some more test with build 99272d6. Initialise a tape: init2.txt According to the help, "parity error" could mean almost anything... How do I do a debug build? I just do "make s2p". I assume there is some switch? |
|
@Pacjunk Please refer to https://github.com/uweseimet/scsi2pi/wiki/Compilation-Instructions for build instructions. |
|
I'm sorry, but given that it takes 3-4 hours to do a build, I'm not constantly checking for updates. I tend to do a rebuild at night and then maybe test the next day. Because of time zone differences, this is when you make your changes so I'm always testing stale code. I will try the debug build. Also you edit your comments and github doesn't send an email for edits, only the original. I managed to get a couple of tape drives working (had to reinstall in another enclosure), then one drive decided to chew some tapes, so this required dismantling to get the tapes out... Anyway, these are the logs: (I tested first without a tape loaded, then with a tape loaded) Sony SDT-5000 : DEC TZ87 in TK50/70 compatibility mode (read only): DEC TZ87 with TK85 tape: I had some problems with that last drive. Not able to initialise tapes for some reason, but seems to read the tape and make all the right noises. |
|
OK, I rebuilt with debug to version 4e1f05a, and tested with the prebuilt simh image. Mounting the tape (successfull): Directory listing of tape (failed): Interestingly, this time I didn't get the generic parity error. The OS reported "file or directory lookup failed. magnetic tape position lost" Is there some sort of utility/option that can "sniff" the traffic going to and from the real drive. Might be interesting to compare it to the emulated drive. |
|
Here is the dump of the file I'm using. It appears to not be in ANSI format. There are 4 files on the tape. VMS054.A, .B, .C, .D. Only the first filename (VMS054.A) is displayed before the error. The filenames are stored next to the data, so you need to wind through the tape to find them. |
|
@Pacjunk I have had a quick look at the latest s2p logs, thank you. I don't think the mount log is required anymore, unless something changes for the worse and mounting fails. In dir.txt I stumble on this: s2p finds the data record at offset 268, which according to the s2psimh dump makes sense. But then it finds a data record also at offset 272, which is wrong. From the dump we can see that the next data record is at offfset 8468. s2p does not appear to skip the data record correctly and is out-of-sync. I will have a look at this. Instead of the s2psimh dump of your file, can you please provide me a download link for the simh file you are using? I can then create a dump myself, and also have the file available for testing. If we assume that the filenames are visibile in the ASCII data of the dump, they should be found. The traffic between your driver and the emulated drive is fully covered by logging on trace level. A tool would not provide more information. Thank you for running the s2pexec commands. The Sony and the DEC drive appear to support reverse-spacing. Since my drive also supports it, this feature appears to be quite common. But that's fine for SCSI2Pi, because I added reverse-spacing about 2 days ago to the issue_100 branch. Regarding the issue_100 branch it's best if you update and re-compile (s2p and s2psimh are sufficient) before running new tests. Re-compiling will not always take very long, because usually only a few files may have changed since the last build. Any news on your web UI issue? |
|
@Pacjunk I have identified why s2p was out-of-sync. Please update and re-build issue_100 and run the directory test again. The s2p log should reveal the change, i.e. s2p should now skip the data record and find the next object at offset 8468 instead of offset 272. I have successfully tested this by sending the SPACE(6) command that was revealing the bug to a simh file with the in-process test tool on my PC: The purpose of this SPACE(6) is to skip up to $7fff data records until a filemark (called tape mark in the simh notation) is found. I guess this is because for a directory listing the data records are not relevant. Filename information appears to be stored in the smaller 80 byte blocks, and a series of these blocks is marked with a filemark. |
|
Only a little testing today - got a migraine! Anyway, the directory listing seems to have worked correctly : I tried to copy the first (small) file off the tape (VMS054.A), but this failed: I think I am using http://www.bitsavers.org/bits/DEC/vax/vms/TK50/aq-jp22f-be_vms_v5.4_bin_1of2_1990.tap for the data. Can't be 100% sure though - the file name is different. I downloaded it a long time ago. |
|
So there is gradual progress. I always need to know the commit ID your tests are based on, please remember this. Your log reveals an issue with the block size: Your system expects a default block size of 8192 bytes. The latest issue_100 branch supports this. Please test again after updating your sources and re-compiling. Ensure that your emulated tape uses a block size of 8192 bytes by adding "-b 8192" to the device parameters when you launch s2p. If you are not sure which file you are using, please re-download it and compare it with your renamed file (e.g. with diff or bcmp), or use a different file. We have to ensure that I use the same file as you for my tests. As soon as it is ensured that we are using the same file, I would be interested in how the directory listing looks like. When you say that you copy a file off the tape, I assume this is equivalent to what "tar xf" does, i.e. extracting the archive contents and writing them to the filesystem? |
|
Sorry, but I didnt get the build number. It was the current version immediately prior to my testing. I have confirmed that the link for the .tap file above is the correct one. The directory listing dump is against that file. The tape contains "save sets", which in unix speak is an uncompressed tar. I believe the default record size for these is 8192, but I believe it is configurable up to 32256. I will check the documentation. I tried to copy one of these "tar" files off the tape, not extract it. This is an allowed operation. Tapes can be mounted in the files11 format (which is what I have done), then the files can be operated on like any other disk files (except no directories). They can also be mounted as "foreign", which is for interchange with other systems or for creating/extracting these save sets directly. The first format is the easiest if you just want to copy a handful of files on or off the tape without having to worry about packing them up into a saveset. The latter is better when you have thousands of files to copy. |
|
@Pacjunk Thank you for providing information on the copy operation. The newer the device, the bigger the supported block sizes appear to be. Current drives support sizes of 512 KB or maybe even more. With SCSI hard drives and MOD SELECT, or with the "-b" option, any size that is a multiple of 4 can be set up to the maximum supported size, which for tape drives is 8192 bytes now. I will try to find out whether your driver tries to change the block size with one of the MODE SELECT commands it sends, but for now using "-b 8192" enforces the required size manually. You can use "git log --abbrev-commit" to get the current commit ID of your checked out git branch, by the way. |
|
Checked the doco. Of course this block size is that within the file container. Standard on disk block size is still 512. (and 2048 for tapes) |
|
@Pacjunk Thank you for checking this. I will try to make the SCSI2Pi block size support even more flexible than it already is. But for your next test the issue_100 branch is already prepared. I checked the MODE SELECT statements: Your driver does not try to change the block size. But I missed that your driver does NOT set the Fixed flag in the READ(6) command, i.e. it explicitly tells the drive how many bytes to read: 8192. Regarding log files, as long as the "dir" test does not fail there is no need to attach a log for the "dir" test. |
|
Except for having to be a multiple of 4 bytes (according to the SCSI standard) the tape drive block size is now completely flexible. When there are read/write requests for any block size (this size can be bigger than the default block size), s2p transfers and reads/writes the data in multiples of the default block size. This is reflected in the log: The bigger the default tape block size, the less read/write operations are needed. But the final result is always the same. |
|
I cannot compile now. I start the compilation and it goes for probably 90 minutes on the one file then: It seems to use most of the CPU and memory. I followed your installation instructions and added: export CXXFLAGS="clang++" I tried grabbing one of the build lines and substituted g++ with clang++ and this worked to build that one module. I don't know enough about makefiles to understand how to fix this, so I just edited the Makefile to change g++ to clang++ I'm yet to see if this works. |
|
Build seems to complete (although it seemed to rebuild nearly everything). I will test later... |
|
Well that went badly - Can't even mount the tape now (can't find the label). Build c826343 Blocksize unset : Blocksize set to 8k : |
I'll check this. Regarding clang++ I just updated the Wiki: Using clang++ instead of g++ compiles faster. I always use clang++ except for building the binary distributions. Note that after changing the compiler running "make clean" is required. |
This should be fixed with the latest issue_100 branch. I could reproduce this problem with the in process test tool and the same .tap file that you are using. For the test I extracted all commands sent by your test and prefixed them with "-c " (for use with the tool) with I removed all commands that are irrelevant for this test like INQUIRY or TEST UNIT READY and set up a sequence of commands in a file "simh.tst" with this content: I run the test on my Linux PC with This way I send the commands in the simh.tst file to s2p. This produces essentially the same s2p logfile as your test, and I can compare the results. In the case of the issue you reported I can see that with the fixed issue_100 branch the results appear to be correct again. Let's see whether your system confirms that ;-). |
|
f46f108 Running the output script on a real tape. This tape is not and cannot be identical, but has the same contents. Script output: s2p output: |
|
If I initialise a blank (zeroed) file, I get the following: As far as I know, the double tape mark signifies the end of the tape, so why is there more data after the double tape mark? You are not writing two tape marks instead of one? |
|
The private end-of-data marker is automatically created by s2p. The two tape marks written by your driver are specific to your driver. s2p may not assume these to always be there and have a particular meaning. tar and drivers for other platforms handle this differently. Regarding a blank file: What I meant is that it should be empty before you initialize it. This means it should not contain anything else but the well-defined data you have dumped above before you run your tests. The two tape marks are created by your driver (init script). I will get back to you as soon as I have analyzed the latest logs. |
|
I did give the build number (beginning of post) and the file I initialised was blank - as I stated. |
|
Yes, I just noticed that. I'm sorry for missing it. |
|
Regarding your test, the sense data returned by your drive include custom information not covered by the SCSI specification. I have tweaked s2p to also return some of these data, let's see what happens. |
|
Build f15e373 I've changed to the Pi 3B - sick of 3 hour+ rebuilds due to low memory. Seems to run s2p faster as well! No real change in latest logs. Attached... |
|
Yes, the transfer rates depend on the CPU and on how fast/how often per time unit the GPIO port can be accessed. The latter is probably even more important than the CPU. I have optimized this, which in one reason for the higher throughput then with the PiSCSI software. Regarding your test I am still unsure what the relevant difference between s2p and your drive is. Can you please run this script with s2pexec against your drive? It is slightly different than the one before wrt getting the sense data. If you can also run it against a different tape drive it would help to find out whether vendor-specific information is returned. |
|
Here you go... It is not easy for me to get the other tape drive at the moment. Maybe tomorrow... |
|
Thank you. Tomorrow (or later) is no problem at all. The reason why I am asking is that your Sony drive returns non-standard data with some error details. Non-standard means that the SCSI specs say that the format used is correct, but a custom one. Reading 90 bytes when only 80 are available is such a case. Testing this with a different drive might provide more insight on the error handling. Regarding your latest log: Can it be that the tape is write-protected? The log says so, but for this script writing is required. The last three bytes are the ones I stumble upon. |
|
Yes, the tape was write protected. I assumed you were doing a test similar to the foreignlist test which was only reading the tape - and I don't want some strange script corrupting that tape. I will re-run the test with a scratch tape. Try this: |
|
Thank you. I'm afraid there was a typo in the script. Can you please try again with: The -n suppresses the automatic REQUEST SENSE, which is run explicitly at the end. |
|
Now it's fine. Again with some custom bytes at the end: I have to figure out what to do next. If this is the only difference between what your drive does with this script and what s2p does, I will temporarily tweak s2p to return excactly the same data, assuming that your driver will proceed then, becasue it receives the same data as will your real drive. |
|
I know I have asked this before, but is it possible to have a program that just records the traffic between the initiator and a nominated SCSI ID? That way you could get a snapshot of the two way conversation involved with a real drive and see if s2p does similar? |
|
As already mentioned, in theory the scsimon tool (PiSCSI) might do that. In practice I doubt that it works, because your computer and drive will likely communicate faster than the Pi can record the data. Or they might even use synchronous transfers, which the Pi would not be able to log at all. But you can give it a try. You will probably get a very verbose log with some (but not all) of the data on the bus. But all commands and data sent are logged in the s2p and s2pexec logs. There is nothing else, i.e. recording the traffic would not reveal more information. Also in practice, so far nothing has been missing, and using scripts with s2p and s2pexec has yielded consistent and comparable results. Everything that has not worked in the past was confirmed/revealed by the logs and scripts. |
|
Just ran it. Don't know what the data means. I think one file is signal waveforms, and html file looks more readable but is severely lacking detail. |
|
I compared the two logs: Except for the additional sense data returned by your drive when 90 instead of the available 80 bytes are read the command executions are identical. This means that either the additional data are relevant, or it is something else that happened before in foreign test, which I have missed. But there is not that much happening in this test, and it stops right after the different error data were returned. If the test fails again, please run the last test script again, twice, against your drive. I want to ensure that these custom data are not something random but always the same. |
|
Build 18a0306 Still a problem: I'm beginning to wonder if this is a tape position issue. When the tape is mounted foreign, subsequent commands don't rewind the tape, but just continue on from the last position. The mount itself does a rewind, and if I force a rewind manually the results are the same. The command to list the tapes' content complains that it has an invalid label, so maybe the "resting place" after a rewind is not correct? Mounting the tape "non-foreign" works fine, but its probably doing everything in one operation. ie rewinding, then loading data, whereas with foreign, the rewind and the read are part of different commands. |
|
The tape position is a good point, indeed. I am not sure what this means for testing, though. Which sequence of scripts or command would be the required one? When you use your real tape, what is the matching sequence of scripts/commands? What I mean is, when you run the foreign mount operation with your drive, which operations were run before? We already have scripts for everything, don't we? |
|
@Pacjunk This ticket is dealing with SIMH file support, but this has already been implemented and we are now dealing with other issues. In addition, I am losing track of all the scripts, logs etc. attached to this ticket. |
|
According to the trace log the last thing the mount/foreign command does is a rewind, so it should be ready for the next command (that we have been analysing). The message the OS gives when trying to list the contents is "tape is not valid ANSI format" - so its obviously trying to look for the ansi field markers e.g. VOL1, HDR1 etc Is there a way to get the trace dump to display the contents of the data read? ie the 80 or 90 bytes. Then we could see if the data aligns with those field markers. I'm testing on a VAXstation 3100 running OpenVMS 7.2. |
|
The trace dumps already display the contents, as long as there is no mismatch between the byte count. Otherwise they cannot display any content because the READ command terminates with an error message. You get the same output when running s2psimh on the same image file: Everything is IMO consistent here, i.e. for a given position on the tape s2p reads and displays the same data as s2psimh. |
|
@Pacjunk There is a new issue_111 branch now, which replaces the issue_100 branch for testing. Please run "git pull" and then "git checkout issue_111". Currently, issue_111 is identical with issue_100. Any further commits will go to issue_111. |
This is a follow-up ticket to #93. The tape device (SCPT) support in SCSI2Pi 4.0 focuses on reading and writing tar files. This has been tested with tar (Linux) and with Gemar (Atari).
In order to support more sophisticated tape drivers, which for instance make use of filemarks and of (reverse) spacing, a different image file format is required. tar files only contain the actual archived data, but not any other objects like filemarks or end-of-data markers, which are used by some drivers, e.g. bacula.
In the ideal case SCSI2Pi can use the same format that is used by open-simh, see https://github.com/open-simh/simh. See http://simh.trailing-edge.com/docs/simh_magtape.pdf for the specification of this file format.
The text was updated successfully, but these errors were encountered: