openBVE Packages

Page 1 of 2 1, 2  Next

View previous topic View next topic Go down

openBVE Packages

Post by leezer3 on Mon May 23, 2016 11:51 pm

Many of you have probably noticed the 'Package Management' tab at the bottom of my openBVE builds.
For a long time, I've thought that we need a package format, which provides basic metadata, dependancies, and is automatically installable with no fuss.

This feature is hopefully now here. At least one member has tried it with reasonable success, and I've done plenty of testing so far, but I'm by no means certain it's bug-free Laughing 

Everything in the Package Manager has been designed as an easy to use 'Wizard', and should be for the most part self-explanatory. There's now a page for this on my build site, here:
http://openbve-project.net/packages/



Linux and Mac users:

Most of my testing has been done on Windows. Packages appear to create and install properly, but there may be some small visual bugs.
Please report any that you see Smile

This is the last 'feature' planned before the first stable release of these builds- See if you can break it Twisted Evil

Edit:

For good measure, this would seem to have be the first package created:
http://www.bveklub.hu/mk48/

Cheers

Chris Lees

http://weww.bvecornwall.co.uk

leezer3

Posts : 747
Join date : 2011-08-23

http://www.bvecornwall.co.uk

Back to top Go down

Re: openBVE Packages

Post by leezer3 on Wed May 25, 2016 3:18 pm

This is version 1.2.0.0 of the 81xx in package format:
http://vps.bvecornwall.co.uk/trains/81xx.zip

I'm only posting it here at the moment, as the package format is technically beta Smile

Changes:

  • OS_ATS has been replaced with BVEC_ATS.
  • Full 3D cab and original 2D cab provided.
  • BR Green and BR Black variants.
  • Passenger and freight consists available.
  • Some sound improvements.


Have also fixed one more bug in the package manager (Installing multiple packages with a website value set in the same session could cause a crash)

Again though, I need someone else to test this- You're much better at finding the things I haven't thought of Razz

Cheers

Chris Lees

http://www.bvecornwall.co.uk

leezer3

Posts : 747
Join date : 2011-08-23

http://www.bvecornwall.co.uk

Back to top Go down

Re: openBVE Packages

Post by phontanka on Wed May 25, 2016 8:26 pm

Chris, I'll sit down to test packages on both Windows and Linux some time this weekend, probably one of the nights. I worked around the zero byte files by adding a comment to them so that their size becomes a couple of bytes at least.

I definitely don't want this to cause a delay of a new release, but could there be a way for more powerful compression, I mean, file sizes are somewhat higher than those of .7z files.

My not so long term plan is to provide everything on BVE Klub as package files for easy installation, but I'm scratching my head a bit because of the increased storage space we will need.

phontanka

Posts : 211
Join date : 2012-05-08
Location : Hungary

http://phontanka.wordpress.com

Back to top Go down

Re: openBVE Packages

Post by leezer3 on Wed May 25, 2016 10:20 pm

Selectable compression types is on my short-term list, don't worry Smile
Similarly, I've been working on adding an additional page to the options dialog today, to control the installation directories etc.

What I don't want to do though is to start adding major layers of complexity until all the kinks are worked out.


Cheers

Chris Lees

http://www.bvecornwall.co.uk

leezer3

Posts : 747
Join date : 2011-08-23

http://www.bvecornwall.co.uk

Back to top Go down

Re: openBVE Packages

Post by MattD6R on Wed May 25, 2016 11:50 pm

I have tested this without too much trouble. I was able to easily create a package using my WIP and also install your packaged 81xx. Though when creating the package when I used a name for the file using the same name as a subfolder it saves in that folder instead of the folder I want. But if I add .zip extension on the end of the file name it saves it in the correct spot. Saving elsewhere without adding the file extension works fine.

MattD6R

Posts : 212
Join date : 2013-06-16
Location : Brisbane, Australia

Back to top Go down

Re: openBVE Packages

Post by leezer3 on Thu May 26, 2016 12:16 am

MattD6R wrote:I have tested this without too much trouble. I was able to easily create a package using my WIP and also install your packaged 81xx. Though when creating the package when I used a name for the file using the same name as a subfolder it saves in that folder instead of the folder I want. But if I add .zip extension on the end of the file name it saves it in the correct spot. Saving elsewhere without adding the file extension works fine.

Hmm, that shows up a rather odd little bug.

The Windows File.OpenWrite API seems to throw an exception if supplied with a filename which is the same as a subdirectory.
What you're seeing appears to be Windows automatically working around it.
Doing some digging, but I get the nasty suspicion that this is one of those annoying little quirks from the DOS days.....


Further note on selectable compression types:
These will likely require WinRAR/ 7-Zip installed to create only.
SharpCompress will decompress anything, but LZMA Zip files (What it's doing at the moment) are the best it can create Smile

Cheers

Chris Lees

http://www.bvecornwall.co.uk

leezer3

Posts : 747
Join date : 2011-08-23

http://www.bvecornwall.co.uk

Back to top Go down

Re: openBVE Packages

Post by MattD6R on Thu May 26, 2016 1:10 am

This was with Windows 10 if that makes any difference.

MattD6R

Posts : 212
Join date : 2013-06-16
Location : Brisbane, Australia

Back to top Go down

Re: openBVE Packages

Post by leezer3 on Thu May 26, 2016 1:40 am

New version is up with various package related fixes:

  • The algorithm used to determine how files should be packaged has had several fixes applied. (Amongst other things, the NWM object directory is no longer thought to be a train....)
  • Added a second page to the options. This allows you to set the installation directories for the three package types, and has a currently non-functional Compression Type dropdown on it.
  • When creating a package, if the output file already exists and is locked, it should no longer crash.


Cheers

Chris Lees

http://www.bvecornwall.co.uk

leezer3

Posts : 747
Join date : 2011-08-23

http://www.bvecornwall.co.uk

Back to top Go down

Re: openBVE Packages

Post by phontanka on Sat May 28, 2016 2:08 am

Chris, I've spent a few hours around packages this night and basically I could create packages without any showstoppers. his was on Windows 10. I faced some issues though, here they are:


  • Package.png: it is not displayed as square, instead at 270*219 pixels. Therefore square images are distorted on the Install Package page
  • On the Select Dependencies page the dropdown doesn't change the view to trains, only routes are visible
  • I tried to uninstall a package but got a message: Unable to uninstall the selected package.
     XML file list missing. The package seems to be still there.
  • On the Select Dependencies page there is no nice name for the header, it just says packages_install_dependencies_header
  • The Add recommendation button seems to add a dependency, not a recommendation


I built a few packages last night on Ubuntu, it worked similarly to the way Windows did, no Linux-specific problems with packages.

These are the packages I've been playing around with:
https://onedrive.live.com/redir?resid=D0BFAA5EB44F4DFF!165&authkey=!AKDR_0_Phiiy2CM&ithint=folder%2czip

phontanka

Posts : 211
Join date : 2012-05-08
Location : Hungary

http://phontanka.wordpress.com

Back to top Go down

Re: openBVE Packages

Post by leezer3 on Sat May 28, 2016 3:00 pm

Will do some proper testing and digging tomorrow Smile

Package.png-
Needs fiddling with a bit more. Ideally, the picturebox wants to automatically take any size of image and represent it in the original aspect ratio. (I'm sure it was set as square, must have got resized at some point and I didn't notice....)


Select Dependancies-
This will be a typo in the code somewhere, or maybe I just haven't added it to the language file....


Add Reccomendation-
Another typo I should think......

XML File List Missing-
Not sure, this is working for me at the moment. The uninstall process needs to be made a little more robust though, as it's clearly deleted the file list, but failed somewhere.....
As an interim fix, just delete the AppData\openBVE\Settings\PackageDatabase folder to flush everything out.
Wild thought- Did you try creating and then installing a package in the same session, or perhaps uninstalling then reinstalling in the same session?

Cheers

Chris Lees

http://www.bvecornwall.co.uk

leezer3

Posts : 747
Join date : 2011-08-23

http://www.bvecornwall.co.uk

Back to top Go down

Re: openBVE Packages

Post by phontanka on Sat May 28, 2016 3:29 pm

Same session thing: not sure, maybe. I tried to pay attention to close and start OpenBVE again between two package operations, but I'm not quite sure.

phontanka

Posts : 211
Join date : 2012-05-08
Location : Hungary

http://phontanka.wordpress.com

Back to top Go down

Re: openBVE Packages

Post by leezer3 on Sat May 28, 2016 11:12 pm

All fixed other than the XML file list error Smile

Still playing with that one, haven't found a way to break it yet, but I'm sure it's in there somewhere.

You obviously shouldn't have to close and re-open the program between package operations, but there are so many re-used variables it's probably the source of the problem somewhere Very Happy

Cheers

Chris Lees

http://www.bvecornwall.co.uk

leezer3

Posts : 747
Join date : 2011-08-23

http://www.bvecornwall.co.uk

Back to top Go down

Re: openBVE Packages

Post by MattD6R on Sun May 29, 2016 1:45 am

When creating a package I got a unhandled window forms exception message when "clear selection" was selected without having any folders added. And I get that saving bug I had before I mentioned if I had the file extension on the end of the file name. After the package (using my WIP) was created it showed the list of files added but as the list didn't fit in the box a scroll bar or a message to indicate that there is more to the list would be helpful. The package came out slightly smaller in size compared to the same files compressed with 7zip.

MattD6R

Posts : 212
Join date : 2013-06-16
Location : Brisbane, Australia

Back to top Go down

Re: openBVE Packages

Post by Quork on Sun May 29, 2016 1:04 pm

Considering this makes LZMA compressed zips only according to Chris, I think I'd check my 7zip settings if I were you. Their help page is quite helpful in finding the best settings!

Quork

Posts : 1030
Join date : 2012-05-05
Age : 25
Location : Frankfurt am Main (Frankfurt on Main), Hessen (Hesse), European Union (Deutschland (Germany))

http://www.parkbahnschmiden.de/

Back to top Go down

Re: openBVE Packages

Post by MattD6R on Sun May 29, 2016 1:14 pm

I just used the default settings (for online backup), and looking at the compressed size it about one third smaller. I didn't notice this as the package uncompressed is only 15 mb. Thanks for your info. Smile

MattD6R

Posts : 212
Join date : 2013-06-16
Location : Brisbane, Australia

Back to top Go down

Re: openBVE Packages

Post by leezer3 on Sun May 29, 2016 10:37 pm

Quork wrote:Considering this makes LZMA compressed zips only according to Chris, I think I'd check my 7zip settings if I were you. Their help page is quite helpful in finding the best settings!

If you want to be absolutely techincal about it, the Sharpcompress library will also create 'plain' zips , tgz and bz2 files.
On the options page, the compression format dropdown will now allow you to select all three of these Smile

You shouldn't be looking at an absolutely massive size increase though, absolutely no more than 10%-
Some quick and dirty tests with the Plymouth route object folder (75.3mb size)

For reference, 7zip on the absolute maximum solid compression gives a filesize of 62.5mb
(I can get away with that, as I've got 16gb of RAM on this box....)


LZMA Zip 65.2mb (+ ~5%)
BZip2 65.7mb (+ ~5%)
TarGZ 64.3mb ( + ~2%)

Your mileage will vary a little depending on how well you've already compressed your textures (I strongly recommend PNGGauntlet on everything)

Actually better than I was expecting.

Phontanka- What sort of size differential are you seeing?
2% is almost not enough to worry about, although I appreciate that if you've got gigabytes of files, this may well add up to quite a bit.


Finally, have fixed the crash and added scrollbars to the boxes in question Smile

Edit:
Some quick tests with SolymarRetro from your test files.

7-Zip, everything maxed out: 49.2mb
TGZ: 69.1mb ( + 40%)
LZMA Zip: 61.3mb ( + 25%)
BZip2: 65.7mb ( +33%)

Frankly though, that one is in *serious* need of a little optimization Razz
Some of the texture sizes are frankly nuts (4096 pixel wide grass texture? From the outline, I'd say it's probably some sort of hill, but that will murder performance), and the power of 2 rule isn't being obeyed very often either, which means that the sim will resize most stuff on load with much longer load times.

Running PNGGauntlet over it (Haven't touched the textures other than converting everything to BMP) reduces the folder size from 129mb to 61.8mb
This then gives us archive sizes as follows:
7-Zip, everything maxed out: 49.3mb
TGZ: 56.9mb ( + 15%)
LZMA Zip: 55.9mb ( + 13%)
BZip2: 55.9mb ( + 13%)

With some image downsizing, I think you could probably get the archive to under 30mb with absolutely no visible quality loss. I'd *guess* 7-zip is probably picking up better on similar areas within images, but don't know.

Cheers

Chris Lees

http://www.bvecornwall.co.uk

leezer3

Posts : 747
Join date : 2011-08-23

http://www.bvecornwall.co.uk

Back to top Go down

Re: openBVE Packages

Post by phontanka on Mon May 30, 2016 12:09 am

4096 - cool! Yeah, maybe some optimization is needed there. I'll take a look at compression again as I have time.

phontanka

Posts : 211
Join date : 2012-05-08
Location : Hungary

http://phontanka.wordpress.com

Back to top Go down

Re: openBVE Packages

Post by MattD6R on Mon May 30, 2016 12:20 pm

When I create a package and select route, replace an existing package and select replace package without having selecting a package I get a  "Unhandled Windows Forms Exception" message.

When creating a package if you don't add a folder to add the package it will continue to the next step. Not changing the version number doesn't give a message to amend it like the other metadata.

With my test using my WIP at the max compression I can do I get a size of 11.5mb with 7zip. Using LZMA in package creation I get it slightly smaller at 10mb. I not sure how this can be but I happy with that as the uncompressed folders is 16mb.

MattD6R

Posts : 212
Join date : 2013-06-16
Location : Brisbane, Australia

Back to top Go down

Re: openBVE Packages

Post by leezer3 on Thu Jun 02, 2016 11:27 pm

MattD6R wrote:When I create a package and select route, replace an existing package and select replace package without having selecting a package I get a  "Unhandled Windows Forms Exception" message.

When creating a package if you don't add a folder to add the package it will continue to the next step. Not changing the version number doesn't give a message to amend it like the other metadata.

With my test using my WIP at the max compression I can do I get a size of 11.5mb with 7zip. Using LZMA in package creation I get it slightly smaller at 10mb. I not sure how this can be but I happy with that as the uncompressed folders is 16mb.

Fixed the first one of those Smile
#2 isn't really a bug:
A common use for 'empty' packages in systems like this is as a pointer to a replacement package where the name/ ID has changed.

To give a quick example:
I've created the package UK_LowerQuadrants.
I now wish to add upper quadrants to the package, but '00s of routes link to the existing lower quadrants download already.
Thus, I create the new package UK_Semaphores, and replace the UK_LowerQuadrants package with a blank package, that depends on UK_Semaphores.
This way, anything linking to UK_LowerQuadrants can continue to do so, but I can provide all future updates in the UK_Semaphores package.

Other changes in today's build related to packages:

  • Text boxes are no longer editable when installing a package.
  • Scrollbars added to the description box.
  • If the database is empty and a user selects the 'Replace' option during package creation, then an appropriate prompt will be shown.
  • If a package uninstall fails, a prompt offering to remove the package from the database will be shown.


It also has one other fix related to bogies (Using a Reversed statement would cause unexpected results), but that's a little O/T for this thread Razz

Cheers

Chris Lees

http://www.bvecornwall.co.uk

leezer3

Posts : 747
Join date : 2011-08-23

http://www.bvecornwall.co.uk

Back to top Go down

Re: openBVE Packages

Post by leezer3 on Fri Jun 03, 2016 2:25 pm

OK, so again I'm only posting this in the packages thread just at the moment Smile

I've been working on a Western for the better part of a year now, and it's finally in a state where it's relatively ready to release.
http://www.bvecornwall.co.uk/wordpress/br-class-52-western/

This train is provided in package format, and requires a nightly from today (03/06/2016) or later, as I've added one further animation variable which it uses.
It's also running a much more recent version of BVEC_ATS, which uses various of the plugin interface changes.

Features:

  • Full 3D cab & externals (No 2D cab)
  • Twin-engine traction modelling.
  • Turbocharger.
  • Relatively prototypical startup sequence and fault modelling- You *will* need to refer to the driver's guide to get this one running Very Happy 
  • Proper bogies (See the other thread)


Still looking for more package related bugs- The next version of the Plymouth route is nearly ready for release, and will provide a real-world example of a package with dependancies.
My initial intention is to post it in this thread, and if nothing crops up with it to progress onto an initial alpha stable build, as opposed to the current nightly situation.

Cheers

Chris Lees

http://www.bvecornwall.co.uk

leezer3

Posts : 747
Join date : 2011-08-23

http://www.bvecornwall.co.uk

Back to top Go down

Re: openBVE Packages

Post by LabRatAndy on Fri Jun 03, 2016 11:01 pm

Packages don't seam to be installed to the correct folder. They are installing to openBVE root folder where ever you have set openbve to find userdata not the userdata sub folder.
ie. C:\users\username\roaming\openbve\Legacy contrent
not
C:\users\username\roaming\openbve\userdata\Legacy contrent

LabRatAndy

Posts : 95
Join date : 2011-08-29

Back to top Go down

Re: openBVE Packages

Post by MattD6R on Fri Jun 03, 2016 11:06 pm

Upon installing your Western if I select install package but before selecting a package I click on the  link to the website I get a Unhandled Window Form Exception message. Once I select the package to install I don't get this crash if I click on the link. It installed without a problem and is very nice train. Great Work. Smile  However when installed the external of the train doesn't work, with just the white box but only for the loco with the externals for the carriages appearing correctly.

When uninstalling if I select train and click uninstall without selecting which train to uninstall, it uninstalls one of them. I have the western and 81xx installed with the 81xx uninstalled when I do this.

It does installs to the AppData folder outside of my Openbve folder which  hopefully is only for the meantime though I know I can change the install location.

MattD6R

Posts : 212
Join date : 2013-06-16
Location : Brisbane, Australia

Back to top Go down

Re: openBVE Packages

Post by leezer3 on Sat Jun 04, 2016 12:16 pm

MattD6R wrote:Upon installing your Western if I select install package but before selecting a package I click on the  link to the website I get a Unhandled Window Form Exception message. Once I select the package to install I don't get this crash if I click on the link. It installed without a problem and is very nice train. Great Work. Smile  However when installed the external of the train doesn't work, with just the white box but only for the loco with the externals for the carriages appearing correctly.
Fixed the exception (LinkLabels would appear to need a special case null check.....)

The white box explanation:
As we found earlier in the thread, SharpCompress doesn't like trying to extract zero-byte files.
The Western uses a null object in a couple of animation sequences (Exhaust smoke)
If an object is missing from an animated file, the whole file fails to load......

That's obviously not at all ideal, it really ought to just display a blank for the missing object.
Going to have a proper bash at the animated object parser in a minute.

Adding the file D1015\External\ExhaustSmoke\null.b3d will make the body appear......

I'll upload a fixed version of it shortly too, but I'd rather fix the root cause Smile

MattD6R wrote:When uninstalling if I select train and click uninstall without selecting which train to uninstall, it uninstalls one of them. I have the western and 81xx installed with the 81xx uninstalled when I do this.
Fixed.

MattD6R wrote:It does installs to the AppData folder outside of my Openbve folder which  hopefully is only for the meantime though I know I can change the install location.

That's a little more of a problem:
There's no real convention for where people keep openBVE content. I know of the following places that are relatively commonly used:
Windows:
Appdata

openBVE Program Folder\UserData

openBVE Program Folder
Linux:
openBVE Program Folder

/usr/share/games/openBVE/

/usr/share/games/BVE/

/home/USER/.openBVE/
OS-X:
openBVE Program Folder

/USER FOLDER/.openBVE/

/USER FOLDER/openBVE/

The only place which I can guarantee will be available and have write privileges on any operating system is the Appdata folder (This resolves to /home/USER/.openBVE/ on Linux or /USER FOLDER/.openBVE/ on a Mac)


Packages don't seam to be installed to the correct folder. They are installing to openBVE root folder where ever you have set openbve to find userdata not the userdata sub folder.
ie. C:\users\username\roaming\openbve\Legacy contrent
not
C:\users\username\roaming\openbve\userdata\Legacy contrent

I don't see a problem there?
If your userdata folder is set for example to C:\Games\openBVE\UserData\ I'm not quite sure why you'd want to create a separate UserData folder within this?
The C:\Users\Username folder is unique on a per-user basis.

That does bring about a slightly nasty thought though-
If two users set the same package installation folder, then things could mess up.
Not sure what can be done about this, will have to think......

Cheers

Chris Lees

http://www.bvecornwall.co.uk

leezer3

Posts : 747
Join date : 2011-08-23

http://www.bvecornwall.co.uk

Back to top Go down

Re: openBVE Packages

Post by leezer3 on Sat Jun 04, 2016 12:44 pm

The null object bug has now been fixed Smile
The Western should now display correctly, albeit with two error messages (Harmless)


Cheers

Chris Lees

http://www.bvecornwall.co.uk


Last edited by leezer3 on Sat Jun 04, 2016 1:10 pm; edited 1 time in total

leezer3

Posts : 747
Join date : 2011-08-23

http://www.bvecornwall.co.uk

Back to top Go down

Re: openBVE Packages

Post by MattD6R on Sat Jun 04, 2016 1:08 pm

With the white box problem I added the null file which sorted it out. However the bogies for the loco are missing with extra bogies following on the back of the train. With the fixed version I get the same problem with the bogies.

As with the content storage I have mine in a folder in the Openbve folder. For the user (especially new users) having it somewhere in the Openbve folder would be make it simple because the user should know where they installed Openbve.  Maybe some setting to set the location but of course the location has to be accessible to the program.

MattD6R

Posts : 212
Join date : 2013-06-16
Location : Brisbane, Australia

Back to top Go down

Re: openBVE Packages

Post by Sponsored content Today at 2:27 pm


Sponsored content


Back to top Go down

Page 1 of 2 1, 2  Next

View previous topic View next topic Back to top


 
Permissions in this forum:
You cannot reply to topics in this forum