code:
		<title>changelog</title>
		<link href="hydrus.ico" rel="shortcut icon" />
		<link href="style.css" rel="stylesheet" type="text/css" />

changelog

version 349

duplicate filter:

the duplicate filter page now has a full-on real-deal file search object to narrow down the duplicate filter, potential duplicate count, and 'show some random dupes' search domains! it also has a 'both files match' checkbox that determines if one of both files of the potential pairs should match the search!

the duplicate filter page has multiple layout changes as a result:

the main management area is now split into two pages--'preparation', for doing maintenance and discovery work, and 'filtering', for actioning the potential dupe pairs found

the 'filtering' page will select by default, but if 'preparation' needs work, its name will become 'preparation (needs work)'

the 'filtering' page now has file search ui and the 'both files' checkbox instead of the file domain button. this search data is saved on a per-page basis

the two pages' status texts are now updated on separate calls which have been rewritten to be asynchronous (with 'updating...' text while they work). both now have explicit refresh buttons to force them to update

the additional non-unknown pair counts listed on the filter area, which were irrelevant to filtering and sometimes confusing, are now gone. it only lists the 'unknown' pair number

the duplicate filter page's help button no longer has the awful 'simple help' entry. the full html help will get a pass in the coming weeks to reflect the new search changes

the duplicate file db code received significant refactoring and improvement to support searching the potential dupe space while cross-referencing the new file search context (and still falling back to the fast code when the search is just blank/system:everything)

misc duplicate file db code cleanup and refactoring

while in advanced mode, you can no longer select 'all known files' file domain for an export folder (and now the duplicate filter page) search context

making a file delete action in the duplicate filter (by hitting delete key or the button on the top hover window, which both trigger a dialog asking to delete one or both) now auto-skips the current pair

.

manage tags:

the manage tags has a new 'siblings and parents' button that will auto-replace incorrect siblings and auto-add missing parents! it works on multi-file selections as well! it gives you a summary yes/no dialog before it fires

the manage tags dialog has a little logic cleanup r.e. siblings and parents and their cog auto-apply options. the auto-application now only applies on add/pend actions

the manage tags dialog has a new cog button option to not trigger 'remove' actions from an autocomplete dropdown or suggested tag input action when the tag already exists

.

the rest:

gave video metadata parsing another pass--it now detects 'hidden' incorrect framerates due to advanced 'skip frame' codec settings and is more accurate at determining frame count and duration, including some fixed offset calculations that was sometimes adding or discounting a few frames

manual video frame count, when needed, is now faster and produces better error text

fixed a critical bug in thumbnail regen that was sometimes potentially looping regen on files with unusual rotation exif information

significant improvements to how the client file manager handles thumbnail identifier information, saving a great deal of time for file import and thumbnail regeneration code of videos

fixed an issue where regenerated file metadata was not propagating up to the ui level in real time

cleaned up some thumbnail cache initialisation code

the 'generate video thumbs this % in' option is moved from the 'media' to 'thumbnails' options page

to simplify code, and in prep for the idle-maintenance-rewrite of this system, the database->regen->thumbnails call is now removed

all three fields of text on serialised pngs now wrap, and they pad a little better as well

added a new option to the 'gui pages' options page to force input text box focus on page changes

fixed a small type issue with the server's session cookie code and some new library versions

version 348

wrote some OR search help to 'getting started with tags' help page

wrote a new multi-reader, single-writer lock object for the client file manager, along with some unit tests for it.

updated the file and thumbnail access and regen and maintenance code to use the new lock. various access is now faster when available and overall safer. there is still work to do here

adjusted file import to be less aggressive about locking, which should reduce some file/thumbnail access lag during heavy imports

the thumbnail space estimate in the migrate database dialog is now more adaptive to the new more flexible thumbnail size system and specificies better that it is an estimate

the client api's /add_tags/add_tags call now collapses siblings and expands parents on an add/pend call. this can be turned off by setting the new optional parameter 'add_siblings_and_parents' to false. the help is updated regarding this and the client api version is incremented to 6

fixed the client api's /add_tags/add_tags call for the 'hashes' parameter, which was failing to parse, and added an accidentally missing unit test to check this in future

the client local services (the booru and client api) now have two new options under their 'manage services' panel: 'support CORS', which turns on cross-orogin support (which is experimental for now, so defaults to False), and 'logs requests', which controls whether your log will be spammed with request reports (this also defaults to False), which should clear up some 100MB+ log hassle for people using the Hydrus Companion browser add-on

hydrus services now respond correctly (albeit sparsely) to OPTIONS requests, and if CORS is enabled, to CORS OPTIONS requests. there are unit tests for this that seem to work ok, but I think we'll need to verify it irl

finished a first version bitmap manager to handle all wx bitmap creation and destruction, including recycling mid-steps

updated all simple wx bitmap creation and destruction calls across the client to use the new bitmap manager, improving stability and saving some CPU

fixed some incorrect button alignment flags that were causing problems for clients set to assert check these values

added a new yiff.party file url class to the defaults that matches a new file attachment format

updated the 'url' content parser so if a parsed url is in the form 'summary_text url', as some booru source fields sometimes specify, the preceding summary text is removed, cleaning up the resultant url

silenced an old server connection lost error that was needlessly loud

silenced the client network engine from additionally log-printing SizeException errors when a downloading file (usually gif) exceeds file import options rules

improved misc window destruction code

updated supported mime list in 'getting started with files' help and website index

misc cleanup

version 347

or search:

under construction OR predicates now present at the top of the regular tag results list, prepended with 'OR: ', and skipping default selection

this new OR line is enter-able, which will present it as-is, rather than adding new preds

hitting escape on a 'search' tag input box that is empty but has an under construction or predicate will cancel the or pred

hitting escape on a 'search' tag input box otherwise should more reliably kill its focus when the dropdown is a float window

improved OR search efficiency significantly with dynamic OR search triggering based on other search predicates. OR searches including negated '-tag' components should be massively faster when paired with non-OR tag or file search predicates

I believe I fixed a search issue that would sometimes return insufficient results when OR preds are mixed with certain other combinations of tags

improved reliability of some thumbnail refresh calls

cleaned up a bunch of OR handling ui code

.

the rest:

after previous weeks' experiments, wrote new double-layer thumbnail loading system--now too-small thumbs will quckly scale up fuzzily straight to screen, and then in the coming seconds, the nice regenerated full-size thumb will be made and drawn in place as ready. it presents much faster and looks better, but there is some cleanup to do here that I will tackle next week

all local file trashing events now record a context-appropriate deletion statement such as "Deleted from Media Viewer." this value is recovered in 'deleted' import status 'notes'. You will mostly see 'Unknown deletion reason.', for files deleted before this new system, but it will populate with appropriate info over time

fixed a search optimisation that was not cross-referencing with file domain, meaning for instance that bare system:rating calls were returning since-deleted files

upnp management window now uses new listctrl

cleaned up some old custom page-naming code

added a 'data' debug call to clear out all cached thumbnails and force an instant ui thumb reload

fixed the trash bmp misalignment, ha ha

removed e-hentai login script from the defaults, since this testing script is not appropriate for new users

dejanked some media viewer video transitions by cleaning up animation bar rendering and smoothing out video buffer initialisation

cleaned out some surplus subprocess wait calls that were hanging some systems on various 'open externally' calls

fixed multiple syncing problems with 'synchronise' export folders that produce files with subdirectories. subdirectory structures should now be synced correctly and empty folders deleted

export folders that collapse multiple file results to the same duplicated name should, after the next run, do less overwriting to this same name

if an export folder or the regular export dialog makes a file destination path that is above the chosen directory (e.g. if the path starts with ../ or ..\), the export job will error out with an explanation

big manual file exports _should_ be politer to the ui and cause fewer hangs

doing page tab drag and drops may have less post-drop ui jank on linux, continued feedback would be appreciated

moved 'reason' handling for all content updates to its own area, which neatens many content update data handling issues

fixed petitioning a tag via a shortcut, which had bad reason handling

fixed an issue with committing pending ipfs items that was overchecking service permissions

fixed some remaining bad wx code in the unit tests

misc file status reporting cleanup

version 346

or search:

extended the search predicate object to handle more OR stuff

extended the tag list to handle list objects that have multiple colours

extended the new OR search predicate to report multiple text-snippet-and-colour pairs based on sub-predicates

extended tag search input to handle prototype OR predicate creation--hold shift when you enter the tag, and it'll start an OR chain. shift-enter continues the chain, enter alone completes it

fleshed out the predicate unit tests to cover more of this

wrote unit tests for OR search predicates. it seems good!

improved some search logic to apply system preds better in certain edge cases and spend less CPU on OR-search-only searches

.

thumbnails:

thumbnails will now queue for load in a more intelligent order based on estimated difficulty to regenerate, which will tend to put more thumbs on screen faster

the decision to regenerate a thumbnail from source is now tempered by how different the current thumbnail is from what is desired--the more similar the two sizes, the more (randomly) likely the client will decide to just use the current (resized) this time. this smooths out change-lag while limiting the number of really fuzzy thumbs you get. feedback on how this works IRL would be appreciated--it uses some voodoo distribution polling to figure it out, which I can definitely tweak

improved visual quality of thumbnail scale-up optimisations

fixed an issue where a multipage thumbnail grid would incorrectly recalculate the new virtual height after a thumbnail size change event, leading to a bit of invalid extra scrollspace (with noclip rendering errors) at the bottom

the thumbnail right-click menu's reparse files entry is now extended to a new 'regenerate' submenu with three options: reparse file and regen thumbs (the old action), force regen thumbs, and regen thumbs if wrong size!

the new 'regen if thumbs wrong size' action sends how many thumbs needed resize up to the popup window, as well

moved some old thumbnail regen code responsibility out of the db and into the files manager

cleaned out some old redundant file/thumbnail code

cleaned and refactored a bunch of general image handling and resizing code

.

the rest:

fixed some bad serialisation code that was making file search objects set their 'include current tags' value to false/true on interleaving loads. on this update, all 'include current tags' values are blanket reset to true

fixed an issue that was drawing animation canvases pure white on various media update events

extended manage urls dialog to support multiple files when launched from a selection of thumbnails. there is a warning in this case, noting that only gallery-style urls are appropriate to be added to multiple files

manage urls dialog now supports multiple selections, including shift-select, and accepts delete key presses for easy mass deletion

when you ask the database migration dialog to move some files, it now pops up a confirmation dialog that also asks if you would like to limit the max time for the job as 10, 30, or 60 minutes

improved file permission setting code across the program to be more sensible for non-Windows

if you are a non-Windows user and were hit with directory permission problems last week on the thumbnail update--which resulted in the rxx directories not being deleted--the update this week will attempt to do the delete again, this time correcting the now missing execute permission bit. if it finds outstanding rxx directories to delete, it will give a popup beforehand summarising the situation and giving you a chance to bail out

fixed yet another problem that was stopping client api url requests from finding the correct page by name

when a client api url request includes fixed tags, these tags should now propagate in all scenarios where the single url produces multiple files

updated sqlite dll and console for windows

misc fixes and cleanup

version 345

or search:

set out a plan to achieve some simple conjunctive normal form (e.g. (blue eyes OR green eyes) AND (blonde hair OR red hair)) OR search support

started work on the object extension and search code to support this search in a very basic (and likely inefficient-for-some-scenarios) way--we'll work on this as we discover the most common inefficiencies

.

thumbnails:

the client no longer uses both 'master' and 'resized' thumbnails--it uses a single, smarter thumbnail

only the 'txx' thumbnail directories (formerly referred to as full-size) are now used, and the thumbnails inside will regenerate and scale themselves as needed on demand (and will be careful to not save changes to disk when when their source file is non-local)

the old 'rxx' 'resized' thumbnail directories are no longer referred to anywhere in the code or ui

the old 'rxx' thumbnails directories will be permanently (i.e. no recycle bin) deleted on update. this is a big job, and you will be prompted on update before it happens

if you have migrated your db to put 'resized' thumbs on an SSD but not the formerly 'full-size', you will want to recheck the 'migrate database' dialog once you have booted and set a new thumbnail override to move the txx directories over

due to the smarter thumbnail, 200x200 is no longer the hard limit for hydrus thumbnails! you can now set up to 2048x2048

all file storage location information is now stored directly in the client db (rather than the options object), which should make for more easily export/importable options in future and improve manual fixing as needed

added more thumbnail-resizing related popup spam to file report mode

fixed a windows-only issue that was making the migrate db dialog close after a file move event concluded

updated database migration help for new concepts and ui

cleaned up some misc storage code

.

the rest:

fixed a problem in the client api with fetching file identifiers from file_ids

fleshed out 'help my db is broke.txt' with more specific clone recovery examples

fixed import support for a variety of single-frame music webms

fixed an edge-case preview viewer initialisation bug that was trying to draw the canvas before any media was set

network report mode now states url classes of urls about to be parsed

misc small fixes and cleanup

version 344

final v1.0 client api polish:

added optional 'show_destination_page' arg to '/add_urls/add_url', defaulting to False, to control whether an URL-add will select (i.e. jump to) the destination page in the ui. this changes the default behaviour for this command

simplified the routine that finds or creates a watcher or url import page and fixed a bug in the api that was not creating new pages when destination_page_name was specified

some misc cleanup

fixed fetching file_metadata by hashes

fixed the client api help regarding file_metadata response example tags

client api version is now 5

.

the rest:

psd support added! because of this format's potential multi-layer complexity, it will not render natively, but width and height are parsed. it is treated as 'application/x-photoshop'. PSB is also recognised and treated as psd

added a 'open_known_url' shortcut to the 'media' shortcut set that lets you quickly open URLs for files. if there is one recognised known url, it will be launched, and if there are multiple, a list with all known urls will appear to select which one you want

animation scanbars now show an x/y current timestamp! it includes millisecond timings and even works for variable framerate gifs. whether to show a second/shadow caret for timestamp position on variable frame rate is a new discussion to have

fixed an issue where animations would sometimes not resume animation for several seconds after a big scanbar drag

when the thumbnail manager cannot produce a thumbnail due to a storage error (like a missing file), it now only puts up a single, more informative error popup on the first problem. subsequent errors are printed silently to the log. (these errors tend to come in en masse, so this cuts down on spam and error-related ui lag that was making loading a bad session difficult)

improved error reporting when an upload pending command would fail due to service non-functionality--it should now give a popup with error info imediately, rather than obscured through the login system

added temp_dir parameter to the client and server that will override which temporary directory the program will use

cleaned up how no_daemons and no_wal mode are handled internally

no_wal mode now has to be called from the command parameter, the no_wal file hack in the db directory no longer works

missing ffmpeg errors now prompt the user to check if it is installed

searching for numerical ratings should now work for files that were rated when the service had a different number of stars (ratings now searches in 'bands' rather than exact values)

reduced the min height of the new import files frame's list

doubled the decompression bomb test to permit files up to ~179 megapixel, we'll see how it goes

misc cleanup

version 343

client api:

fixed an int/str type mismatch issue with service_names_to_actions_to_tags in /add_tags/add_tags in the client api that meant that argument was not working

fixed up some last /get_files/search_files stuff

added /get_files/file_metadata

added /get_files/file

added /get_files/thumbnail

added help and unit tests to reflect the above

updated client api version to 4

.

the rest:

the list of paths in the manual file import dialog is now sortable. this order will be preserved in regular and 'add tags' ok events for this dialog. it has a new '#' column so you can return to 'parse' order if desired

animation and static image windows in the media viewer canvas are now recycled through media type transitions, making for slightly smoother browsing between mixed media

increased aggression of media viewer image prefetch

added support for 'MM' Tiffs

fixed webm mime parsing for webms with no audio (these were falling back to mkv)

improved the error reports when a serialised png fails to import

the mass-open-urls popup is now pausable as well as cancellable

fixed several recently broken ui unit tests

misc old code cleanup

some misc test controller/constant refactoring

version 342

added support for webp import. it does not yet support animated webps, which, if the local platform supports, will import like apngs used to: just the first frame

added support for tiff import. it works ok for 24bit and 8bit (monochrome) tiffs, but I am not sure how well it will do with 48bit

both webp and tiff should work on the duplicate files system

improved webm detection to include opus audio (previously, these files were falling back to mkv)

fixed an issue where unusual formats with duration but no frames or frames but no duration were being sorted and otherwise presented incorrectly

improved autocomplete job cancelability. this job can now cancel much faster on large jobs, meaning typing searches with large result sets will hit less CPU and return faster on subsequent keystrokes

_all_ of the complicated 'copy url' commands from the thumbnail right-click->known urls menu are now available on the 'open' submenu! if there is more than one url to open (e.g. 'open all of these files blahbooru post urls' on a selection of 50 files), you will be presented with a yes/no dialog to confirm, and it will open one url in your browser every second (with a cancellable popup if num_urls > 5)

by default, system:everything is now hidden if its total files is >10k. you can force it to always show under options->default system predicates

the gallery downloader's list's status column now shows gallery status (deferring to active file status) when appropriate and shows 'done!' when all work is complete

after working back and forth with a user, I _believe_ the linux similar files >0 distance search crash is finally fixed

fixed sorting by media views/viewtiming with collections

a single-selected collection right-click now shows total media views for all files in the collection! you can now see how long you have been viewing an artist!

fixed an issue that lead to export folders not running on always-on clients as often as they should

updated the gelbooru 0.2.5 file page parser to pull rating tag from the correct location (previously, it was pulling from what appears to be a site-wide 'mature' browser hint)

improved memory cleanup stability when animations and other parts of the video rendering pipeline are deleted--this _may_ fix some rare crashes

increased animation rendering aggression overall and particularly in 'future' of frame buffer

if a video renderer that is asked to start some way into the video fails to render anything, it will now fall back to trying to render from the beginning. this is slightly hacky atm and leads to out of phase rendering frames, but it is better than an error

added a '--no_db_temp_files' launch parameter that will force the client or server to return to the recent old behaviour of exclusively using memory for journalling. this is useful if your temp directory is small and/or your available ram is very large. if running in this mode, the client will attempt to check available memory (instead of free space on your temp dir) before performing very large transactions

with the new lighter-weight update transactions, the client now tests for less free space for journalling before running repository update processing

added /get_files/search_files to the client api, which does the first half of file searching. it allows tag search (including -tag negation) and system inbox/archive. since the second half, which will fetch file metadata, is not yet in, this can't do anything interesting yet

updated help and unit tests to support this, client api version is now 3

some misc refactoring

version 341

client api:

added /add_tags/add_tags, which does several kinds of tag content updates

added /add_tags/clean_tags, which shows how hydrus will handle potential tags

added /add_urls/associate_url, which allows you to associate urls with files

added 'destination_page_name' to /add_urls/add_url, which will choose which destination watcher/url importer to place the url (or create a new one with that name)

updated client api version to 2

updated client help and unit tests for the above

added a linked contents to the client api help

improved some server error handling, mostly moving 403s to more correct 400s

improved how missing parameter 400 errors are reported from the server vs deeper keyerrors that should be 500

.

the rest:

tag repository update processing now saves progress to disk every million rows or every minute, whichever comes first. this reduces journaling bloat, improves recovery when the process quits unexpectedly, and makes for significantly faster cancel when requested by the user

when processing duplicates and copying/merging/moving ratings, the 'source' file will now also overwrite the 'destination' file's rating if that destination rating is lower (previously, the rating would only go over if the dest had no rating set)

added a new 'thumbnail experiment mode' under help->debug->gui. this will load fullsize thumbs and resize them in memory, please see release post for more details

reduced menubar replacement flicker while, I believe, keeping and strengthening recent menubar indexing stability improvements

the tag autocomplete dropdown will now always embed (instead of floating) in non-Windows

when data seems non-decodable, the fallback encoding format is now that given by chardet, rather than utf-8

improved serialisability of some pending tag data

watchers can now hold and pass on fixed pending tag data

gallery log objects can now hold and pass on fixed pending tag data

file import objects can now hold and action fixed pending tag data

hard drive imports now store their paths-to-tags info in this new format, directly in the file import objects

improved some url-import page drop-target-selection logic

improved error reporting when dropping/api-adding urls

adjusted some url import workflow so big 'already in db' download lists should work a bit faster

attempting to start the program with some external database files but not the main 'client.db/server.db' file will now cause a boot-fail exception with an explanation before any stub db files can be made

tightened up some hydrus service login-capability-testing code that was previously stopping certain error states from recovering promptly, even on a force account refresh, while the service was maxed on bandwidth

fixed a source of linux CRITICAL logspam related to several common dialogs

improved ui stability on boot when file folders are missing (particularly for linux)

improved stability for the various async tasks on the duplicates processing page, particularly for linux. I am not sure I got everything here, but it is definitely better

did some more misc stability improvements, particularly in various boot fail scenarios

completely removed an ancient and janky focus catcher widget from the main gui frame

now various db caching is improved on the python side, removed a sqlite instruction to force temp information to always stay in memory--hot data returns to staying mostly in memory to start and then spools to disk if the transaction gets too large

fixed approx bitrate sorting for malformed video files with explicitly '0' duration

daemon_profile_mode now spams some more info about export folders

fixed an issue that meant client db maintenance was firing its jobs too aggressively, regardless of idle status

updated windows build to cv 4.0

misc refactoring and fixes

version 340

client api:

fixed up some api permissions object stuff so that /verify_access_key response can always serialise correctly

fixed the 'add_url' api call's instability

the API will now always return JSON on 200. anything else should be presumed to be raw text

'/api_version' now returns JSON, and after talking with users, it will now start incrementing with every api change. it remains 1 just for this week

'/request_access_permissions' now returns JSON

'/add_url' now results JSON on success with more info, 403 on failure

'/get_url_info' now returns the 'normalised_url' in the response JSON

added '/get_url_files', which returns 'url_file_statuses', listing known hashes and file import status for that url

added '/add_files/add_file', which can import a file from a path or bytes

added '/add_tags/get_tag_services', which will return info on the client's tag services

updated client api help to reflect the above changes and fleshed out the intro a bit

fixed the client api permissions enum values in the help, which I somehow transcribed wrong first time

updated the client api tests to check the above

refactored client api tests to be neater and in their own file

.

the rest:

fixed the page of pages close bug

added a downloader for nijie.info to the client defaults (it needs a login)

updated danbooru file page parsers to get 'rating' tag

added gelbooru 0.1.11 parser for future application

fixed an issue that was stopping advanced content updates from fully copying all the desired mappings in the transaction

added a semi-hacky checkbox to 'options->files and trash' that will delay all new file/thumb requests for 15s after the computer resumes from sleep (useful if your files are on a NAS that takes a few seconds to reconnect on wake)

wrote some more graceful fallback decoding handling code that attempts original assumed encoding and 'utf-8' if different and returns the one with the fewest '�' replacement characters

the network engine and the ffmpeg info parsing now use this new 'safe' decoding, so even if a site has borked bytes or the video file has unexpected Shift-JIS title metadata, it'll still go through, albeit with some question marks

moved some more old daemons to the new job scheduler, deleted some old daemon code

improved some daemon job wake and shutdown code

wrote a proper upnp manager object and improved all-around reliability of the auto upnp-service-mapping code

simplified the upnp check code so it now only ever checks/does anything if the respective services actually want upnp mappings. surplus mappings are now wiped immediately on service update

fixed upnp mapping fetching to cope with ipv6 results

improved some memory clearing code to deal with some semi-stubborn objects

improved some 'iterate through this giant list of single numbers from the db without using a lot of memory' code and applied it to the autocomplete cache regeneration routine

improved menubar stability, both in finding menus and swapping them out

if a serialised json object fails to load from the db, this is now caught, the bad object deleted and written to a new file in the db dir, and all logging info captured along with an explanatory popup thrown on screen. so, if a subscription fails to load, it will now be extracted so that a subsequent subscription edit/run will work with the remaining good objects. in the case of backed-up objects (gui sessions atm), reattempting the load should restore the next most recent backup

fixed an issue with login script validation when the given credentials have surplus ( key, value ) pairs to the script's credential definitions

fixed two login invalid cookie error handling bugs

maybe made some dupe filter searching more stable

fixed a py2 datatype issue that made the client unbootable when updating the client from <296

the client now pauses to nag and moan about backups if you try to update more than 15 versions in one go

slightly sped up discord bugfix file drag and drops and expanded file limit up to 25 files/200MB

added experimental secret discord bugfix dnd mode checkbox

improved how html parsing deals with some unexpected bad tag data

turned on primitive high-dpi support for OS X. let me know if it fixes any blurry issues on retina displays

wrote a new 'ui test' under the debug->gui menu to help catch common-action bugs that slipped through weekly work

improved how the test code does some wx/ui stuff, but also broke some more and ran out of time to clean it up--this is an ongoing project

improved how some text import line splitting works

misc fixes

version 339

client api:

wrote some ui to handle client api permissions adding and editing

wrote a 'catch a permissions request' mini-dialog for external api permissions adding

wrote api calls:

GET

/api_version

/request_new_permissions

/verify_access_key

/add_urls/get_url_info

POST

/add_urls/add_url

and made a new 'client api' help page to describe in detail what these do

wrote fairly comprehensive unit tests for the new client api

refactored a bunch of 'hydrus network' specific stuff away from general server code that the client api now uses

neatened up 401 vs 403 error handling across the program, and replaced some clientside error handling that was inelegantly borrowing these network errors

deleted very old prototype file/thumbnail client server fetch code, which was no longer in use

.

misc and bug fixes:

added a 'clear ratings' button to the ratings service 'review services' panels. it can clear out ratings for deleted, non-local, or _all_ files

the '3 downloaders are working, is it ok to close the client?' message is now folded into the 'confirm client exit (auto-yes in 15s)' dialog. this merged dialog will still appear for users who have the regular confirm client exit dialog turned off (and still auto-yeses in 15s)

the file url downloader now reports 'downloading file' and 'importing url' text status separately

fixed a typo bug from last week that was breaking asc/desc ratings service sorting

fixed a typo bug from last week that was stopping manage import folders from opening

fixed a typo bug from last week that was breaking setting upnp port on the local booru/client api service management panel

the advanced file reparse-and-re-thumbnailing routine now correctly moves a file to its new extension if its mime changes (e.g. png->apng, or webm/mkv distinctions)

the client file manager now silently detects and auto-repairs instances of missing files where the file actually does exist, just with the wrong extension

fixed a url parsing issue that was normalise-mangling url classes with no path but some query parameters

the network engine now uses utf-8 decoding when no specific encoding is set (previously ISO-8859-1)

fixed an ffmpeg video parsing bug when the video included Shift-JIS metadata. it should work for other unusual encodings as well

maybe cleaned up some menubar management code

the filename tagging dialog now uses a notebook for service choice, like the manage tags dialog, rather than the janky old listbook

fixed a py2-to-3 issue with the admin-only 'is server currently busy' check while a backup is running

improved some dialog button event handling. it may completely fix the 'trytoendmodal' issue some users run into

improved some JSON db serialisation error reporting code, trying to pin down an issue several users have seen with session save

improved thread-safety of serialisable objects as they serialise

misc improvements and cleanup

version 338

after talking with some users, put a bit of time into tag autocomplete wolkflow. both read (search pages) and write (manage tags) autocomplete inputs now operate ~asynchronously~, with the tag fetch working on a separate thread. tag jobs can also now cancel at certain checkpoints in the tag search process if overwritten by a new request. therefore, a variety of tag lookup scenarios _should_ be less painful. this change was executed in a semi-inelegant way, so please report if you encounter bugs from fast typing etc...

I also improved some of the "I hit enter before results were in" code as a result of this. not sure I have it totally nailed, so please give feedback on errors here

wildcard search tags now have an explicit '(wildcard search)' after their label

taglists that have an attached page of media (basically the 'selection tags' box and the 'active search predicates' up top) now have a right-click menu entry to 'select files with (all these tags)' and, if more than one tag is selected, 'select files with (any of these tags)'! This is pretty neat in action, so give it a go!

added a 'sort by approx bitrate' file sorting option that does a basic filesize/duration so you can filter out dense gifs and other short-but-big vids a bit easier. anything without a duration is shoved to the 'smallest' side

some of the file sort options now default to their respective 'biggest' first, see how you like it

folded in updated gelbooru file page parser (fixing the 403 errors that just appeared) and added searches for gelbooru user favourites and pools

if a gallery or watchable url ends up in the file processing queue (and hence fails), it now says the believed-to-be url class name in the error, which should help some false positive url class matching debugging

fixed a focused-file selection issue that meant preview viewtime was frequently counting in file viewing statistics even while a page was not currently in view

fixed the local booru review service panel, which wasn't fully deleting shares when the button was clicked

wrote a service object and basic server skeleton for the client api (basically refactoring the existing local booru code). client api now appears in manage and review services and can boot and present the normal hydrus browser welcome page at '/'

the client api and the local booru now have an 'allow non-local connections' checkbox! defaulting to off and on respectively

updated a wx-thread call function to more safely and universally deal with instances where the responsible window died before the call could be made

removed some old experimental crypto code that isn't used any more--pycryptodome is no longer needed to run the program from source

improved some misc client service code

deleted some old unused code

misc cleanup

updated to ffmpeg 4.1 on windows release

updated to sqlite 3.26 on windows release

updated to wxpython 4.0.4 on all platforms

version 337

fixed another couple of unicode encoding problems with the logging and profiling code

the logger now sticks a unicode BOM at the top of new log files to help text readers guess the utf-8 encoding

fixed musical webm import when the video stream has no stated duration but the audio stream does and typically stretches out a 'single frame' video

fixed some 'max size' download file size testing

'waiting on bandwidth' statements on network job controls now show the specific network context (like 'web domain: somesite.com') they are waiting on (you might need to hover over to see the tooltip for this)

the downloader easy-importer lain image is now clickable to launch a file selection dialog

if you are in advanced mode, the manual file export dialog now lets you export symlinks with a new checkbox. this is experimental, so if you are interested, give it a brief test and let me know how it works for you

duplicate content merging now applies to pending as well as current tags

the duplicate filter now counts pending tags when saying which file has more tags

advanced content updates now _copy_ both current and pending tags. the other actions now state what they do more clearly

stopped printing long server error text on 304 and 404, where actual response content is uninteresting

removed wx import that accidentally came into server space due to 2to3 check

improved the path-fixing code that helps environment construction when launching external programs from non-windows frozen builds

fixed a critical pubsub processing bug that kicked in at a certain stage of client shutdown. this should stop the post-shutdown-processing memory explosion certain users were seeing and should stop any ui jank in the last 0.2s of the program for everyone else

improved some other shutdown memory cleanup that was sometimes leading to double-log-printing of exit statements

did a full pass over the daemon scheduling code. it now reacts more responsively to various shut down situations

reduced db disk cache aggression significantly and added more memory maintenance to the cache population process

fixed an issue where subscriptions were not promptly responding to shutdown events

fixed an issue where some delayed network jobs (e.g. while all network traffic is paused) were also not responding to shutdown events

added a 'pubsub report mode' debug mode for simpler pubsub review

the db is now less redundantly spammy on certain behind the scenes update notifications

wrote a first version of the client api manager and permissions handling objects

misc fixes, cleaned some shutdown code

version 336

fixed an issue where the numerical rating control was coercing all clicks to either the minimum or maximum allowable rating (e.g. 3/5 stars wasn't working)

fixed some text file and process i/o, which was handling some unicode decoding/encoding incorrectly. it now mandates utf-8 in all cases

fixed a referral url encoding problem that was stopping pixiv from downloading when the gallery page url had kana/kanji characters (from a search term)

fixed a str vs bytes issue when loading the filename tagging panel

fixed the delete button on the filename tagging quick namespaces panel (the edit and delete buttons are also now 'live' and will disable when nothing is selected)

improved some json dump deserialisation code

fixed a data-sorting issue that would appear with certain parsers in the edit parsers panel

improved video metadata parsing, fixing an issue when the video has a 'title' row containing inconvenient data

fixed some hex character processing for system preds

added an advanced check item to the gallery downloader cog icon menu that will 'bundle' multiple query-pastes to the same single gallery downloader (this is helpful if you are pasting a whole bunch of md5 queries in one go and would rather one downloader work through them sequentially than 50+ separate ones blat your CPU simultaneously)

the different kinds of importer worker threads now have several limits on the max number that can be working at once, to stop accidental ui overload when a hundred or more are in memory and all want to work at once (like after a big paste event or resuming after computer sleep). during periods of heavy import activity, the importers will now naturally space themselves out to smooth out the spike. the limits are hardcoded for now, let me know if it noticeably bottlenecks your situation

made some menubar update code a bit less complicated and reduced how often it'll spam during heavy update

the 'what to do?' buttons that appear in manage tags sometimes on a tag action got a simplification pass and are now on the new dialog system

simplified my new dialog code significantly, clearing out redundant code and classes and pushing all okable/cancellable/vetoable closing checks through one single method

wrote some new help.txt in the db dir about hanging startups

version 335

important:

hydrus now runs completely and exclusively on python 3!

for users who are updating, the client has special install instructions for just this week:

if you are a windows or linux user who extracts to install, you will have to delete your old install's files (but keep your db folder!!!) before installing/extracting the new version so there are no 2/3 dll/so conflicts (don't delete your db folder!)

if you use the windows installer to install, this v335 installer will do the clean install for you! there is absolutely no way this could go wrong, so no need to make a backup beforehand :^)

if you are an os x user, I am now only releasing the client in the app. furthermore, the default app db location is now ~/Library/Hydrus (i.e. /Users/[you]/Library/Hydrus). you will have to move your existing db to this location to update, and thereafter you'll just be replacing the app in Applications!

if you try to boot a non-clean mixed 2/3 install, the client will try to recognise that and give an error and dump out

please check the release post for more detailed instructions here

.

semi-important:

the db password feature may be one-time broken for unusual keyboard languages, so failures this version will be forgiven with an appropriate error message explaining the situation. feedback from ???? ????? lads appreciated

I may have fixed the issue some linux/os x users were having launching external programs, including OS ffmpeg (it was a child process environment issue related to pyinstaller)

although I did most of my devving here on py 3.6, the client seems to run ok on 3.5. I doubt 3.4 will do it, if you mean to run from source

I moved from the old pycrypto to the new pycryptodome, so users who run from source will want to get this. I also dropped some libraries

.

misc bug fixes:

fixed the 'load one of the default options' button on manage tag import options when a set of default options is orphaned by a deleted url class

removed some popup flicker related to long error messages

fixed some parsing testing ui error handling

cleared up some bad text ctrl event handling that could sometimes cause a recursive loop

listctrls should now sort text that includes numbers in the human-friendly 2 < 10 fashion

cleaned up some bad external process calling code and improved how child process environment is set up

finally figured out the basic problem of a long-time nested dialog event handling error that could sometimes freeze the ui. I may have fixed it in one case and will keep working on this

.

boring details:

ran 2to3 to auto-convert what could be done

updated environment to python 3

went over a whole ton of unicode encoding/decoding manually to update it to python 3

removed all the old tobytestring/tounicode calls in favour of new python 3 handling

fixed all the file io to do bytes/str as appropriate

corrected a bunch of / vs // int/float stuff

fixed up twisted, which has some str/bytes stuff going on

fixed all the listctrls to deal with column sorting None values amongst ints/strs

fixed png export/import, which had some fun bytes/bytearray/int/str issues

updated the swf header parsing code to py3 (more str/bytes stuff)

misc float/int fixes

fixed up some http.cookies handling, which has changed in py3

improved some ancient loopback connection code that was only really checking to see if ports were in use

cleaned up a bunch of now-invalid parameter tuples that 2to3 helpfully marked

numerous misc other refactoring and so on

updated the new network engine to now decode non-utf-8 responses correctly based on actual response header

removed some old py2 manual http multipart code

removed the old py2 'matroska or webm' parsing py, replacing it with some direct ffmpeg format inspection

replaced all % formatting with the new .format system. I will slowly move to this rather than the current endless concatenation mess

deleted some more misc old code

tightened up some spammy network error reporting

converted all /r/n to /n in my environment project, ha ha ha

the ui seems to better support rarer unicode characters like ??

updated some of the install/update/backup help for all this, and some misc other stuff as well

fixed misc bugs

version 334

wrote a system:file viewing stats to comprehensively search the new viewing stats--it _should_ also be synced with the exact current values

but for system:everything, inbox, and archive, which remain where they were, system predicates are now sorted alphabetically!

added a _database->file viewing stats_ menu that lets you suspend file view tracking and clear all records permanently

mr. bones now welcomes all users under the help menu

fixed mr. bones's confusion at those who have yet to board the ride

also mr. bones now makes sure to get the latest file viewing stats

moved confirm trash/archive from _options->gui_ to _options->files and trash_

moved a bunch of 'pages' related stuff from _options->gui_ to the new _options->gui pages_

added an option to options->gui pages to change the number of session rolling backups

subscription popups now provide an x/y query progress string in their popup text

the edit subs/sub panels are now a bit shorter by default and the edit sub has its own frame position data, under 'edit_subscription_dialog', and remembers its last size and position by default

fixed an issue where some dupe watcher urls (like url and url#12345) were not being correctly merged on a mouse drag and drop watcher-import

the client will now print up to 512KB of server error info to the log (previously 4KB)

removed the youtube download prototype--if it returns, we'll do a proper youtube-dl solution. as a result, pafy is no longer needed to run the client

network report mode now shows more network error information

gave the 'getting started with subscriptions' help page a complete pass. it now reflects the new system and has up-to-date advice based on my new experience

wrote a 'logins' section to the bottom of the 'getting started with downloading' help page

misc fixes

version 333

added a first version of file viewing statistics! the client db now keeps track of how many times a file is loaded in the preview and full media viewers, and for how long!

you can see the media and preview stats on any single media right-click menu. there are multiple options for how this displays, including hiding it completely, under options->media

viewing stats update as they happen! (although viewtime typically only updates on the end of viewing. I'll likely make this more live, especially if I end up showing this info in the media media viewer)

you can now sort files by total media views/viewtime!

mr. bones's wild ride continues, as well

deleted the old 'file list' way of updating in-ui media objects in favour of a long-planned global media cache. there is now only ever one active copy of any particular media, and all data-level updates need only occur once on that single copy. this saves a bunch of CPU, memory, and overall hassle behind the scenes! various search results/lookups for media already loaded elsewhere now load super fast!

tag siblings refresh is quicker and less memory heavy thanks to this as well

furthermore, the complicated tag changes from tag repository processing and advanced content updates are now reflected immediately in the gui on the job's completion! (as long as you have fewer than 10k files open, ha ha) previously, these required a search refresh to show the results

the file sort choice dropdown on all pages is now sorted alphabetically. it has always been a mess picking what you want from here, so let's see if this helps!

tag and rating sort options are now listed as 'tag:' and 'rating:' respectively

fixed some misc file sort choice code, which was failing to keep certain defaults in certain situations

fixed the tag import options' new 'load from defaults' button to correctly load the tag blacklist

the keyboard icon on the media viewer's top hover window now permits activation of current/default shortcut sets under submenus. it now also omits these entries if no custom shortcut sets exist

cleaned up some of the hover_window-canvas interaction code

fixed some long-time sperg-out  buffer-drawing when changing position in a long video

the database->backup actions are now hidden if the current db has non-default file/thumbnail locations. for now, in these cases, only a custom backup is appropriate

fixed some ancient repository admin code that fetches summary account info given an account key

the filename tagging dialog now has a much shorter listctrl by default, so should fit better on smaller monitors

fixed the 'review session cookies' dialog's clear button, which was not deleting sessions after clear. it now also wraps the operation in a yes/no confirmation

version 332

the client serialisation system now supports multiple rolling backups!

client sessions (like the 'last session' that typically loads on boot) are now automatically backed up to ten times in rolling backups! you can review and append the backups if you need to recover from _pages->sessions->append session backup_, where they are listed by their timestamp!

when the client closes, an additional 'exit session' is now saved. this differs to 'last session', which is overwritten every x minutes, and is now available (especially with the new backup) for various error recovery situations

gallery import pages now have a little cog icon to control if new import queues will start with files and/or galleries paused. these states persist through a session reload

tag import options that are 'defaultable' now have a button to let you load in a specific default, so you can easily quickly edit in a one-time slight alteration of the default rather than having to create everything from scratch

under options->downloading, you can now set the 'delay' times on gallery/watcher network error, subscription network error, and subscription other error, now defaulting to 90mins, 12hours, and 36 hours respectively

attempting to launch a client with db version > software version now spawns a blocking messagebox on pre-launch informing the user of the risks and advising task manager force-kill of the process

did a little cleanup on the new tag id database cache and merged some other, older semi-laggy tag-fetching code to use the same system

wrote a similar 'file id' database cache for caching file hashes and also merged some old hash-fetching code into it. a variety of file operations are now significantly faster

export folders can now delete files from the client after export. the edit panel will warn you on selecting this and oking the panel. it can't be set if the export type is 'synchronise'.

fixed the edit export folders dialog's old buttons, which were semi-working due to some recent update work

when checking for file integrity, you can now choose to export a .txt file listing all the missing files' known urls, so you can try to recover by feeding them all back into a new url downloader!

the physical file deletion process is now a little simpler and deals with larger jobs in smaller batches, no longer hogging the file read lock the whole time. clearing a large trash should no longer hang other media loads

adding ngugs in the 'export downloaders' dialog now attempts to add the respective gugs as well

fixed an issue where the listctrl would accept certain kinds of duplicate data and hence confuse its indices

fixed all the add buttons on the export downloader panel to exclude all items already in the list when figuring out what additional objects to add

the 'scroll thumbs at this rate per tick' option now _rounds_ the pixel result, rather than always _floor_ing it

wrote a new mass-selection database routine that should reduce memory footprint of autocomplete regeneration. if it works out, I will use it in some other places

improved the errors when a network job that cannot wait fails on an invalid login, and added a separate error when it refers to a hydrus service

fixed a small typo bug when trying to auto-add url classes and parsers

updated an old deprecated checkboxlist call

clarified the integrity section of 'help my db is broke.txt'

misc fixes

version 331

added a 'do login now' button to the manage logins dialog. it only enables when the selected logins are active and not invalid and so on, and will ok the dialog and queue up some login attempts, which will make report popups as normal

'review session cookies' panels now support drag and drop cookies.txt import! cookies.txt importing will also handle errors a bit better and report total number of added cookies

the 'review session cookies' panel now defaults to not showing sessions with zero cookies. a new checkbox controls this

login scripts can now be rolled into easy import pngs! should work for export and import just like the other objects (although they won't be auto-added based on domain in export dialog)

brushed up some of the 'change login script' code--particularly, it now puts login scripts that have matching domains first in the selection list, for easier selection

after striking a reckless bargain with a daemon from the database-plane, system:num_tags now runs significantly faster and produces accurate tag counts even when searching over multiple tag services that have duplicate tags. if this works out, the immaterial beast promises greater gains for similar jobs with no possibility of anything going wrong

prototyped a new tag cache in the db that affects (and should speed up) many tag fetching routines. let's see how it goes

added complete, global proxy support for the new network engine! there are new options under options->connection (with some explanation text) to handle it. if pysocks is installed, socks4/5 proxies are also available!

updated the e-hentai.org login script to the new one on the github. your existing mappings for e-hentai.org _should_ all be updated right. exhentai.org is likely too difficult to properly support in the current system

the different panels where you enter system predicate information now all run on the new panel sizing system--if you have had problems with these, please let me know how they size now!

added a '4channel thread' url class to support watchers for the new 4channel sfw domain. it works for now, but let's see if their api changes when the split actually happens

the list right-click menu on gallery import and thread watcher panels now has three options to show combined importers' files--presented, new, and all. it also now shows the files (more smoothly) in the same page, clearing any existing highlight.

misc ui improvements

updated 'running from source' in help

version 330

login:

added a proper username/password login script for hentai foundry--double-check your hf filters are set how you want in your profile, and your hydrus should inherit the same rules

fixed the gelbooru login script from last week, which typoed safebooru.com instead of .org

fixed the pixiv login 'link' to correctly say nsfw rather than everything, which wasn't going through last week right

improved the pixiv file page api parser to veto on 'could not access nsfw due to not logged in' status, although in further testing, this state seems to be rarer than previously/completely gone

added login scripts from the github for shimmie, sankaku, and e-hentai--thanks to Cuddlebear and any other users who helped put these together

added safebooru.donmai.us to danbooru login

improved the deviant art file page parser to get the 'full' embedded image link at higher preference than the standard embed, and only get the 'download' button if it looks like an image (hence, deviant art should stop getting 140MB brush zips!)

the manage logins panel now says when a login is expected to expire

the manage logins dialog now has a 'scrub invalidity' button to 'try again' a login that broke due to server error or similar

entering blank/invalid credentials is now permitted in the manage logins panel, and if entered on an 'active' domain, it will additionally deactivate it automatically

the manage logins panel is better at figuring out and updating validity after changes

the 'required cookies' in login scripts and steps now use string match names! hence, dynamically named cookies can now be checked! all existing checks are updated to fixed-string string matches

improved some cookie lookup code

improved some login manager script-updating code

deleted all the old legacy login code

misc login ui cleanup and fixes

.

other:

sped up tag searches in certain situations (usually huge inbox) by using a different optimisation

increased the repository mappings processing chunk size from 1k to 50k, which greatly increases processing in certain situations. let's see how it goes for different users--I may revisit the pipeline here to make it more flexible for faster and slower hard drives

many of the 'select from a list of texts' dialogs--such as when you select a gallery to download from--are now on the new panel system. the list will grow and shrink depending on its length and available screen real estate

.

misc:

extended my new dialog panel code so it can ask a question before an OK happens

fixed an issue with scanning through videos that have non-integer frame-counts due to previous misparsing

fixed a issue where file import objects that have been removed from the list but were still lingering on the list ui were not rendering their (invalid) index correctly

when export folders fail to do their work, the error is now presented in a better way and all export folders are paused

fixed an issue where the export files dialog could not boot if the most previous export phrase was invalid

the duplicate filter page now has a button to more easily edit the default merge options

increased the sibling/parent refresh delay for 1s to 8s

hydrus repository sync fails due to network login issues or manual network user cancel will now be caught properly and a reasonable delay added

additional errors on repository sync will cause a reasonable delay on future work but still elevate the error

converted import folder management ui to the new panel system

refactored import folder ui code to ClientGUIImport.py

converted export folder management ui to the new panel system

refactored export folder ui code to the new ClientGUIExport.py

refactored manual file export ui code to ClientGUIExport.py

deleted some very old imageboard dumping management code

deleted some very old contact management code

did a little prep work for some 'show background image behind thumbs', including the start of a bitmap manager. I'll give it another go later

version 329

login:

the login manager is fully turned on! hentai-foundry click-through and pixiv login now occur fully on the new system

wrote a Deviant Art login script for NSFW downloading--however, it only seems to work on a client that has done some logged-out downloading first (otherwise it thinks you are a robot)

updated the DA file page parser to only NSFW-veto if the user is currently logged out

wrote a danbooru login script for user prefs and special files if you have a gold account

wrote a gelbooru 0.2.x login script for user prefs

pixiv recently(?) allowed non-logged in users to see sfw content, so the login script is updated to reflect this. the login script doesn't detect a failed login any more, so I will revisit this

logging in in the regular order of things now makes a temporary popup message with the overall login status and final result. ~it is cancellable~--and if cancelled, future login attempts will be delayed

logging in in the regular order of things now prints simple started/result lines to the log

deleted old network->login menu and related code such as the custom pixiv login management. gdpr click-through is now under downloaders

subscription login errors will now specify the given login failure reason

subscription login tests will now occur at a better time, guaranteeing the sub will be correctly saved paused if the test fails

login errors will now always specify the domain for which they failed

testing a login script on a fresh edit login script dialog now pre-fills the alphabetically first example domain

the login script test ui now restores its 'run test' button correctly if the test is abandoned early

misc improvements to login error handling and reporting

.

other:

any texts across the program that ellipsize when they are too thin to display what they have will now tooltip their text (this most importantly includes the status on the network job control, which will now display full login problem info)

the copy button on manage tags goes back to copying all if no tags are selected

the remove button on manage tags now removes only selected if some tags are selected. it still removes all if none are selected

the remove button on manage tags is now wrapped in a yes/no dialog (as is hitting the delete key on the list's selection). this can be turned off under the cog button

filename tagging panels now support directory tagging for the last, second last, and third last directories. the related code for handling directory tagging is cleaned up significantly

the export files panel now lets you delete the files from the client after export. this value will be remembered, and if on will prompt a capital letters warning on export, either via the button or the quick-export shortcut

in manage tag parents, where there are multiple parents in a pending action (either by importing via clipboard/file or by putting multiple parents in right-hand box), the action will now be treated as one transaction with one 'enter a reason' confirmation!

in manage tag siblings, when multiple 'better' values are pended in one action via a clipboard/file import, they will now be treated as one transaction with one 'enter a reason' confirmation!

.

misc:

added a new url class that api-links .gifv-style imgur links so they are downloadable like regular imgur single media pages

the pixiv manga page url class now redirects to the new api, so mode=manga pages should now be drag-and-drop importable and generally downloadable if you have any still hanging around in any queues

clients now come with an additional danbooru parser that fetches the webm version of ugoiras

after discovering a pdf that ate indefinite 100% CPU while trying to parse, I have decided to stop pulling num_words for pdfs. it was always a super inaccurate number, so let's wait for a better solution at a later date. hydrus hence no longer requires pypdf2

fixed an issue with monthly bandwidth estimates rolling over to the new year incorrectly

in an attempt to chase down a duplicate files content move/copy bug, the duplicate action content updates got a bit of cleanup work. if you have noticed duplicate actions not copying tags/urls, please let me know the exact process in the ui, including services and merge options, you went through

tag lists should now update their sibling appearance correctly after a tag siblings dialog ok--previously, they were checking for new sibs too early

tag siblings and parents should now refresh their data more efficiently when spammed with new data notifications (this usually happens janitor-side, which approving dozens at once)

copy queries/watcher urls on the download pages' lists' right-click menus no longer double-spaces the copied texts (it just does single spaces)

fixed an issue where certain initialised watchers were erroring out when asked to provide next-check time estimates--in all cases, null timestamps will be dealt with better here

misc tag parents/siblings ui code cleanup

wrote some code to catch and report on an unusual dialog dismissal error

version 328

wrote test ui for edit login script panel

the login system now works and is turned on, although the legacy hardcoded pixiv and hf logins remain in place. it will not do anything very new this week--it is strictly only for advanced users to experiment with for now

cleaned up some messy network code

all subscription and hydrus jobs will no longer wait indefinitely on an invalid login--they will cancel immediately

network jobs will report a bit more info when they are cancelled

subscriptions will now attempt to test login validity before and during file downloads and syncing. if they fail, the sub will pause and stop work and a message will be presented to the user

made a 'thumbnails' options page and moved some things to it

added thumbnail border and margin to that thumbnails page! you can even set 0 border and/or margin and it works

fixed up a heap of bad thumbnail drawing code that didn't work with thicker borders

the tag sibling and parent dialogs now have suggestion buttons in their 'give a reason' dialogs! if your petitions are simple and fit into one of these categories (which is most of them), please use these buttons as they will let janitors (e.g. hydrus dev for the PTR) process them in batches, in fewer clicks

manage tags dialog's checkboxes and advanced buttons are now wrapped into a cog icon! remove/copy/paste buttons are compacted and put on the same row!

manage tags dialog's copy button now only copies selected, not all tags

manage tags dialog now uses the new sizer. some components are smaller by default but will eat up spare pixels better

misc manage tags code cleanup

added 'paged file import queues' to the network->pause menu. this will pause any, hdd, url, simple, gallery, or watcher page from processing its file import queue. it is a bit hacky and will take up to 30s to unpause unless you joggle the respective downloader to wake it up--see how it goes!

added a similar 'gallery searching' to that menu, to pause any url, simple, or gallery page from actioning its gallery-side queue

and 'watcher checking' for watchers

fixed a stupid bug that was causing false-positive and _mostly_ harmless errors for certain pixiv and artstation multi-page downloads

fixed an issue where multi-page data was being mis-metadata'd (mostly, this meant thread watchers were giving the last filename tag to all files, and I think similarly getting the latest source time) due to a mistake in some recent de-duping code

fixed the new pixiv file page api parser to parse source time, which I must have accidentally deleted previously

fixed a no-expand bug in my new sizer when in horizontal orientation

fixed a small bug when making an easy-import downloader png and cancelling the add url class dialog

misc fixes

version 327

login stuff:

finished off some login script data stuff

fleshed out how login credentials and other linked data is stored in the login manager, including script link recovery when the script changes but name does not

improved some initialisation login validation error handling

improved login failure validation error handling

wrote a dialog panel for managing login credentials and reviewing validity and so on

a heap of related session and login tie-in/fix-up work

the login manager will now save changes to the db. it will get the HF and pixiv scripts on db creation/update, and if you have a pixiv login, the login system will pre-fill that info and 'activate' the script (although the login manager will not fire any login scripts yet--if so configured, it'll just delay on a polite error message)

.

other stuff:

with the subscriptions' new more liberal syncing logic, the periodic file limit will now only pop up if the sub does not see any already-seen files

to give more buffer for the new syncing logic, file import caches will now store 250 entries minimum on compaction (was 100 previously)

subscription merging now lets you choose the primary subscription into which the other subs will be merged

cancelling a subscription merge action mid-merge is now safely nullipotent

post urls that use subsidiary page parsers (such as the new pixiv manga parser) will now correctly insert (rather than append) their manga urls into the file import cache

removed a couple of places where urls could accidentally be duplicated in a file import cache

cleaned up some areas where successful file import objects were presumed to have file hashes when they might not (this was causing errors when importing urls that split into multiple url children, like pixiv manga, while also having 'additional tags' set)

updated tag censorship, parents, and siblings dialogs to the new panel system

tag censorship, parents, and siblings panels now use a notebook instead of the layout-borked listbook

tag parents and siblings panels now use the new small-resolution-friendly sizer, are more tight by default, and expand more neatly

refactored a bunch of tag ui code to clientguitags

the client video renderer will now deal with videos with (invalid) duration of 0 more gracefully

finished the 'getting started with downloading' help page, sans the login stuff

bit of other help work

version 326

login:

finished the new login objects. they can deal with multi-step single second-level domain login problems, can pass variables from step to step, and use cookies as success verification

wrote an ton of ui for the new login objects, now under network->downloader definitions and network->logins. it is not 'active' yet, but advanced users are invited to check it out. there is no good test ui yet, which I think I'll have to figure out in the coming weeks

wrote a first attempt at HF and pixiv replacement login scripts--please try importing from defaults on the manage login scripts dialog and look through them to see what I am going for. once the system is flipped on and we are happy these work, I'll remove the old hardcoded legacy login stuff

when a network job that needs a login cannot login, it now waits (rather than bombing out completely), presenting the related error, and checks again every 60 seconds

if a network job thinks it can login but fails to generate a login process, the network engine now catches the error safely and recovers. the job is put on hold as above

.

subs:

the subscription 'have we caught up to where we were before' test is now more complicated--rather than just stopping after five 'already seen' urls are found, it now only stops if at least the _last_ five contiguous urls of the page are already seen. this will catch more late-tagged files that get inserted out of order

fixed the 'get quality info' button on edit sub panel to only get the current selection, not all queries wew

subscriptions can now optionally publish/present their files to a specific label! this is a great way to merge multiple subs to the same final landing page

.

layout:

after a long time thinking about it, wrote a new custom boxsizer that handles resizing multiple expanding items of different reasonable min size by expanding them _beyond minimum size_ by their proportion, rather than forcing them all to have total proportional width/height. I expect to polish this and apply it in multiple locations around the program where tall things were being too tall because something else was forcing it to be (the management panel on the left of most pages was terrible at this, causing a giganto taglist just because the upper panel was tall as well).

changed my custom boxsizer (the box with a bold header) to the new custom boxsizer, so it is all over now--please report any bad layouts you see

in an effort to improve layout, the manage tag parents and siblings panels' preview boxes have shorter minimum height--it will get a bigger layout overhaul soon

.

bigger misc:

thanks to work of user kourraxspam on the discord, fixed the pixiv downloader to use a more stable api and added pixiv tag search

watchers and gallery imports now have a list right-click menu entry to show all selected importers' files in a new page! use this to clear out a bunch of finished queues all at once!

the tag right-click menu now offers 'open new search pages for each in selection' if multiple tags are selected--this will open three search pages each with one tag, as opposed to the original entry, which would only open one page with all three

the edit nested gug panel now uses a checklistbox rather than the menu to select gugs to add, which is more reliable and allows for multiple selections

sped up autocomplete tag fetches' tag sibling integration--irl this may be a reduction in total a/c search time of approx 33%

page parsers will now generate next gallery urls absent any file/post urls if the only type of url they can generate is gallery urls (so a meta-gallery-search like board->threads that only generates subsidiary gallery pages will now work, whereas before it never could because it was missing post urls)

the gallery log now provides a shorthand way to restart and resume failed searches from its right-click menu (if the most recent log entry failed)

'try again (and allow search to continue)' reattempt jobs will now generate next page urls even if no new urls are found (which can happen if a search stopped due to the file limit exactly lining up with the number of files found, for instance, so a reattempt finds nothing new)

gallery downloaders will now specify their 'delay work for a bit' error states in the ui. this usually means 'could not connect', which has a 4-hour timer (I'll prob add a scrub delays button here at some point)

the watcher will now show its 'delay work for a bit' error state in more places in the ui

added a 'media' shortcut 'export_files_quick_auto_export', which will open the export files frame and give you a quick yes/no to confirm you want to export as set. if yes, it will export. then it will close the frame

added a 'show a "N" to short import summaries' option to options->downloading, which will extend the typical 'x/y' status string to 'x/y - zN' for z 'new files' (as opposed to already in db)

improved how the video parser estimates frame rate--it _should_ fix some of those low-framerate, low-framecount slideshow-vids where at current they render everything in a rush and then sit on the last frame for ten secs

.

smaller misc:

network report mode now reports url_to_fetch and parser-to-parse-with info

when the server fails to accept a file upload due to a file parsing issue, it now prints the hash of the file in the error

if the client sees a possible file hash in a server error message from a file upload, ~it will try to show that file in a new page~

fixed an issue where wildcard searches were not finding results if the search text included the normally discarded characters [](){}"'

fixed some domain handling for localhost and other undotted network names

content parsers will now only launch with permissable content types, which for the legacy 'lookup scripts' scripts system means only tags and vetoes, and for the new login system means only temp variables and vetoes

as compaction now happens automatically on sync, removed the 'compact' button from edit subs panel

an unusual network error related to hydrus update files sometimes being cut off mid-stream is now glossed over silently, with the download reattempted after a delay

the initial gui session load now occurs after a 0.25s delay--let's see if it cleans up some initial layout issues some users have had

maybe fixed an odd dictionary-initialisation error related to tag siblings/parents dialog boot

ruggedised against an unusual bandwidth load bug

gave some of the index help a pass

did most of a 'getting started with downloaders' page in the help--I'll finish it next week

updated discord share link to https://discord.gg/3H8UTpb , which should not expire

some listbox add/edit code cleanup

some listctrl delete code cleanup

misc help work

misc cleanup

version 325

added a 'show a popup while working' checkbox to edit subscription panel--be careful with it, I think maybe only turn it off after you are happy everything is set up right and the sub has run once

advanced mode users will see a new 'get quality info' button on the edit subscription panel. this will some ugly+hacky inbox/archived/deleted info on the selected queries to help you figure out if you are only archiving, say, 2% of one query. this is a quickly made but cpu-expensive way of calculating this info. I can obviously expand it in future, so I would appreciate your thoughts

subscription queries now have an optional display name, which has no bearing on their function but if set will appear instead of query text in various presentation contexts (this is useful, for instance, if the downloader query text deals in something unhelpful like integer artist_id)

subscription queries now each have a simple tag import options! this only allows 'additional tags', in case you want to add some simple per-query tags

selecting 'try again' on file imports that previously failed due to 'deleted' will now pop up a little yes/no asking if you would like to first erase these files' previously deleted file record!

the watcher and gallery import panels now have 'retry failed' buttons and right-click menu entries when appropriate

the watcher and gallery import panels will now do some ui update less frequently when they contain a lot of data

fixed the new human-friendly tag sorting code for ungrouped lexicographic sort orders, where it was accidentally grouping by namespace

downloader easy-import pngs can now hold custom header and bandwidth rules metadata! this info, if explicitly present for the appropriate domain, will be added automatically on the export side as you add gugs. it can also be bundled separately after manually typing a domain to add. on the import side, it is now listed as a new type. longer human-friendly descriptions of all bandwidth and header information being bundled will be displayed during the export and import processes, just as an additional check

for advanced users, added 'do not skip downloading because of known urls/hashes' options to downloader file import options. these checkboxes work like the tag import options ones--ignoring known urls and hashes to force downloads. they are advanced and should not be used unless you have a particular problem to fix

improved how the pre-import url/hash checking code is compared for the tag and file import options, particularly on the hash side

for advanced users, added 'associate additional source urls' to downloader file import options, which governs whether a site's given 'source urls' should be added and trusted for downloaded files. turn this off if the site is giving bad source urls

fixed an unusual problem where gallery searches with search terms that included the search separator (like '6+girls skirt', with a separator of '+') were being overzealously de/encoded (to '6+girls+skirt' rather than '6%2bgirls+skirt')

improved how unicode quoted characters in URLs' query parameters, like %E5%B0%BB%E7%A5%9E%E6%A7%98 are auto-converted to something prettier when the user sees them

the client now tests if 'already in db' results are actually backed by the file structure--now, if a the actual file is missing despite the db record, the import will be force-attempted and the file structure hopefully healed

gallery url jobs will no longer spawn new 'next page' urls if the job yielded 0 _new_ (rather than _total_) file urls (so we should have fixed loops fetching the same x 'already in file import cache' results due to the gallery just passing the same results for n+1 page fetches)

in the edit parsing panels, if the example data currently looks like json, new content parsers will spawn with json formulae, otherwise they will get html formulae

fixed an issue with the default twitter tweet parser pulling the wrong month for source time

added a simple 'media load report mode' to the help debug menu to help figure out some PIL/OpenCV load order stuff

the 'missing locations recovery' dialog that spawns on boot if file locations are missing now uses the new listctrl, so is thankfully sortable! it also works better behind the scenes

this dialog now also has an 'add a possibly correct location' button, which will scan the given directory for the correct prefixes and automatically fill in the list for you

fixed some of the new import folder error reporting

misc code cleanup

version 324

downloaders:

after adding some small new parser tools, wrote a new pixiv downloader that should work with their new dynamic gallery's api. it fetches all an artist's work in one page. some existing pixiv download components will be renamed and detached from your existing subs and downloaders. your existing subs may switch over to the correct pixiv downloader automatically, or you may need to manually set them (you'll get a popup to remind you).

wrote a twitter username lookup downloader. it should skip retweets. it is a bit hacky, so it may collapse if they change something small with their internal javascript api. it fetches 19-20 tweets per 'page', so if the account has 20 rts in a row, it'll likely stop searching there. also, afaik, twitter browsing only works back 3200 tweets or so. I recommend proceeding slowly.

added a simple gelbooru 0.1.11 file page parser to the defaults. it won't link to anything by default, but it is there if you want to put together some booru.org stuff

you can now set your default/favourite download source under options->downloading

.

misc:

the 'do idle work on shutdown' system will now only ask/run once per x time units (including if you say no to the ask dialog). x is one day by default, but can be set in 'maintenance and processing'

added 'max jobs' and 'max jobs per domain' to options->connection. defaults remain 15 and 3

the colour selection buttons across the program now have a right-click menu to import/export #FF0000 hex codes from/to the clipboard

tag namespace colours and namespace rendering options are moved from 'colours' and 'tags' options pages to 'tag summaries', which is renamed to 'tag presentation'

the Lain import dropper now supports pngs with single gugs, url classes, or parsers--not just fully packaged downloaders

fixed an issue where trying to remove a selection of files from the duplicate system (through the advanced duplicates menu) would only apply to the first pair of files

improved some error reporting related to too-long filenames on import

improved error handling for the folder-scanning stage in import folders--now, when it runs into an error, it will preserve its details better, notify the user better, and safely auto-pause the import folder

png export auto-filenames will now be sanitized of \, /, :, *-type OS-path-invalid characters as appropriate as the dialog loads

the 'loading subs' popup message should appear more reliably (after 1s delay) if the first subs are big and loading slow

fixed the 'fullscreen switch' hover window button for the duplicate filter

deleted some old hydrus session management code and db table

some other things that I lost track of. I think it was mostly some little dialog fixes :/

.

advanced downloader stuff:

the test panel on pageparser edit panels now has a 'post pre-parsing conversion' notebook page that shows the given example data after the pre-parsing conversion has occurred, including error information if it failed. it has a summary size/guessed type description and copy and refresh buttons.

the 'raw data' copy/fetch/paste buttons and description are moved down to the raw data page

the pageparser now passes up this post-conversion example data to sub-objects, so they now start with the correctly converted example data

the subsidiarypageparser edit panel now also has a notebook page, also with brief description and copy/refresh buttons, that summarises the raw separated data

the subsidiary page parser now passes up the first post to its sub-objects, so they now start with a single post's example data

content parsers can now sort the strings their formulae get back. you can sort strict lexicographic or the new human-friendly sort that does numbers properly, and of course you can go ascending or descending--if you can get the ids of what you want but they are in the wrong order, you can now easily fix it!

some json dict parsing code now iterates through dict keys lexicographically ascending by default. unfortunately, due to how the python json parser I use works, there isn't a way to process dict items in the original order

the json parsing formula now uses a string match when searching for dictionary keys, so you can now match multiple keys here (as in the pixiv illusts|manga fix). existing dictionary key look-ups will be converted to 'fixed' string matches

the json parsing formula can now get the content type 'dictionary keys', which will fetch all the text keys in the dictionary/Object, if the api designer happens to have put useful data in there, wew

formulae now remove newlines from their parsed texts before they are sent to the StringMatch! so, if you are grabbing some multi-line html and want to test for 'Posted: ' somewhere in that mess, it is now easy.

version 323

wrote first version of the new downloader easy-import drop-panel. you drop downloader-encoded pngs on it, and it maybe asks you a question and jumbles its way through auto-importing all the required data to the client

extended this file import to do some cleverer 'example url merging' when parsers are otherwise dupes, rather than spamming similar dupes on import

wrote first version of the new downloader export panel. it takes gugs, url classes and parsers, and predicts sensible sub-objects to include to make functional downloaders, and bundles it into one png

fleshed out help for the new easy import/export system

the client now slows down gallery and watcher processing when the network engine is under heavy load, aiming for no more than 50 jobs in system at once. the solution is a bit hacky for now, but it should alleviate the deadlock issue when there are ~180+ simultaneous gallery/watcher network jobs pending

the multi-watcher panel's list of watchers now supports right-click menu to copy/open urls and pause/play files/checking

the multi-downloader panel's list of downloaders now supports right-click menu to copy query texts and pause/play files/searching

added a 'derpibooru tag search - no filter' GUG that disables the default derpi no-explicit-files rule

added basic gfycat support to default client--drag and drop any typical video page, and it should import ok

fixed the canvas/hover window tag sorting discrepancy--all tags are now sorted with the same code, and the media view sort order should be the same as your default sort order (although in this case incidence has no effect as there are no tag counts)

rewrote the network job control's cog menu to be a bit more dynamic, and added 'override gallery slot requirements for this job' if appropriate

fixed a stupid typo bug in the shutdown maintenance jobs test code that was causing pending repository work to not report right

fixed gallery searches that include unicode characters that end up in the path of the url (rather than the query parameters)

fixed an issue where highlighting a watcher would unpause its checking

generalised the way the new listctrl class can produce right-click menus

fixed some api link calculation that was over-prescribing api link display pairs (this affected the artstation file page url class by default). these pairs are now also sorted in the links dialog

misc png-export improvements to present better with the new easy import/export stuff

the summary texts in the tag filter panel now ellipsize (...), so if the tag filter is complicated, it won't try to boot a superwide edit panel!

the manage subscriptions panel now correctly initially sorts in a case-insensitive way (previously, it was usually sorting A-Za-z, which is different to regular aA-zZ resorting behaviour, so it always sort-flickered after the first edit)

the status bar has a new segment for reporting when the client is 'busy' with different jobs. for most typical usage, it'll just stay blank. let's see how it goes.

fixed mr. bones's wild review when the client currently has no files

punched up the new file report mode to specify full paths where available

improved some misc downloader code

version 322

wrote gugs help

gave url classes and parsers help a pass

wrote e621 html gallery page example help

wrote gelbooru html file page example help

wrote artstation json file page example help

wrote url class links help

gallery logs for the gallery downloader and url downloader now support 'try again' and 'skip' right-click menu for gallery log entries. the try again allows just the one page or also allowing search to continue) so, if a gallery query fails for some reason, you can now try again/continue where it broke. subs/watcher/simple downloader work on more complicated gallery search logic, so their gallery logs will remain read-only for now

all gallery log buttons now support right-click menu to mass-export urls to png or clipboard. non read-only also support import

fixed an issue with gallery searches that rely on both api url conversions and url class next gallery page urls (I think just artstation and tumblr by default) not generating the next page url correctly

improved some misc gallery url processing logic

fixed some issues with gallery url generators with invalid example urls causing problems opening the edit gug and gallery selector panels

fixed an issue where you could only delete a gug if it was in an ngug, ha ha

thanks to a different submission by prkc on the discord, collections now have a _right-click->set collections as groups of alternates_ duplicate action (note the duplicate menu only appears in advanced mode). the related shortcut action duplicate_media_set_alternate_collections is also added

export phrases now support '\' ('/' in linux) in the path export phrase in order to create folders. you should also be able to do \[series]\ to create optional namespace folders. slashes in tags will still be replaced with _

to stop the client sometimes doing laggy vacuum checks every maintenance cycle, vacuums that cannot occur due to limited disk space now will still count as 'done' for the purposes of rescheduling

added 'file report mode' to the help debug menu. This will spam popups as file and thumbnail actions are requested

tightened up some network job status setting to help us debug the 'there are a ton of jobs in network engine, but the three active on this domain seem stalled' issue

wrote a simple 'review threads' panel under help->debug->data actions->review threads. I knocked it together in about ten minutes, and it's likely unstable as hell, but it's pretty neat!

some instances where many file paths are copied quickly (exporting paths to clipboard and drag and drop) no longer do a safety check for file existence, so should be much faster to go. this particularly reduces startup lag for large file drag and drops!

the 'would you like to do maintenance work in this shutdown?' dialog now lists a summary of what it thinks it'll be working on. I _could_ make this more detailed, so let me know how it works for you

tags with numbers should now sort according to the new improved human sorting method--it now shouldn't matter where the numbers are in the tag--as long as the text-and-number-breaks lines up with another tag, they'll be compared each part in turn correctly

fixed some human sorting code for unusual number characters like ?. they will be treated as text, not a number, for now

misc fixes

version 321

downloader overhaul:

the basic downloader overhaul is complete! at this point, any user can create and share the objects required for a completely new downloader! it is still rough in some places, so a round of EZ-import is coming to make adding new downloaders a single easy drag and drop action

rounded out the ngug (nested gugs, which contain multiple gugs) code

updated the edit gug panel to deal with gugs and ngugs on different notebook pages

added a bunch of logic to this panel and backend data handling to deal with missing gugs in ngugs

if an ngug cannot find a gug by its internal identifier key, it will now attempt to fallback to its simple name, and will silently fail if no gug can be found. all gug tracking now uses this 'key first, name later' id method, so downloaders and subs should generally survive gug renames and same-name overwrites

the gallery selector now works in gugs. it has two 'pages', depending on which gugs are set to 'display', and will note if the chosen gug is cannot be found in the current definitions. the gugs have slightly more specific names ('gelbooru tag search', 'hentai foundry artist lookup', etc...) than before

the gallery selector also puts 'non-functional' gugs (i.e. those with no parsable gallery url class) to a third page

moved the gallery downloader gallery and file pipeline completely over to the new system

the gallery downloader will now bundle nested gugs (like hentai foundry artist, which searches both works and scraps) into a single downloader

moved the subs gallery and file pipeline comppletely over to the new system

the subs gallery sync now handles nested gugs (like hentai foundry artist, which searches both works and scraps) in an interleaved manner and make behind-the-scenes checking decisions in a clearer and more logical way

subs should now make correct 'hit limit' stop reason reports and not generate new gallery pages when the current page has exactly enough results to hit the current file limit

artstation artist lookup is now available as a default downloader

newgrounds artist lookup makes a triumphant return. it works pretty well, given how flash and NG has changed since

derpibooru tag lookup is now available as a default downloader. due to unusual search syntax on derpibooru, please enter queries exactly as you would on derpi, using ',' or ' AND ' to separate tags (such as 'rainbow dash,straight')

pixiv now has multiple artist lookup options--either images, manga, ugoira (doesn't work yet!), or everything

the old downloader code is deleted!

the old manage booru dialog is deleted!

'custom' boorus (i.e. new ones you created or imported to 'manage boorus'), cannot be completely automatically updated to the new system. I've figured out a way to generate new gugs and gallery&post parsers, but they will miss url classes to get working again. your custom-booru subs will notice this and safely pause until the issue is fixed. if you rely on custom boorus, please check the release post for info on this--you might like to put off updating

many misc changes and fixes to gugs and overall gallery url handling pipeline

some misc refactoring and concept-renaming in gallery pipeline r.e. gugs

when the downloader tries to import what looks like a raw html file, its error notes will specify this and suggest a parser may be needed

moved the 'media viewer url display' options panel from the manage url match links dialog to the new network->downloaders->manage downloader and url display

this new dialog also hosts a list for managing which downloaders to show in the first list of the downloader selector

.

misc:

gave the video rendering pipeline communication logic a quick pass, cleaning up a bunch of bad code and other decisions. the video renderer should be quicker to respond to various changes in scanbar position, and incidences of the frame buffer suddenly sperging out (usually inexplicably falling behind the current frame position or deciding to regen for no apparent reason) should be greatly reduced if not completely eliminated

the test that stops repository processing if there is not enough disk space now uses half the current size of client.mappings.db for its estimate (previously 1GB) and also tests temp folder location free space (just as the vacuum test does) and reports this nature of the error along with pausing the repo, stopping further attempts

might have fixed another out-of-order dialog close/open event combination during manage tags close->advanced content update open

fixed gallery queries that include '/' (or some other unusual characters) that end up in the 'path' of the url (as opposed to the query). this fixes 'male/female' on e621, for instance

'advanced mode' users now have a 'nudge subs awake' menu entry below 'manage subs'. this simply wakes the subs daemon (which usually only checks once every four hours), in case any subs are due

'db report mode' now reports every db job as it comes in (formerly, it only reported some optimisation esoterica). this makes it a more lightweight version of 'db profile mode' for several debugging tasks

fixed a tiny issue in fetching the 'how boned am I?' stats when the user had zero inbox/everything count

fixed a typo in the default new url class object that was breaking the edit ui panel

highlighted the quiet filename tagging options on edit import folder panel

version 320

clients should now have objects for all default downloaders. everything should be prepped for the big switchover:

wrote gallery url generators for all the default downloaders and a couple more as well

wrote a gallery parser for deviant art--it also comes with an update to the DA url class because the meta 'next page' link on DA gallery pages is invalid wew!

wrote a gallery parser for hentai foundry, inkbunny, rule34hentai, moebooru (konachan, sakugabooru, yande.re), artstation, newgrounds, and pixiv artist galleries (static html)

added a gallery parser for sankaku

the artstation post url parser no longer fetches cover images

url classes can now support 'default' values for path components and query parameters! so, if your url might be missing a page=1 initialsation value due to user drag-and-drop, you can auto-add it in the normalisation step!

if the entered default does not match the rules of the component or parameter, it will be cleared back to none!

all appropriate default gallery url classes (which is most) now have these default values. all default gallery url classes will be overwritten on db update

three test 'search initialisation' url classes that attempted to fix this problem a different way will be deleted on update, if present

updated some other url classes

when checking source urls during the pre-download import status check, the client will now distrust parsed source urls if the files they seem to refer to also have other urls of the same url class as the file import object being actioned (basically, this is some logic that tries to detect bad source url attribution, where multiple files on a booru (typically including alternate edits) are all source-url'd back to a single original)

gallery page parsing now discounts parsed 'next page' urls that are the same as the page that fetched them (some gallery end-points link themselves as the next page, wew)

json parsing formulae that are set to parse all 'list' items will now also parse all dictionary entries if faced with a dict instead!

added new stop-gap 'stop checking' logic in subscription syncing for certain low-gallery-count edge-cases

fixed an issue where (typically new) subscriptions were bugging out trying to figure a default stop_reason on certain page results

fixed an unusual listctrl delete item index-tracking error that would sometimes cause exceptions on the 'try to link url stuff together' button press and maybe some other places

thanks to a submission from user prkc on the discord, we now have 'import cookies.txt' buttons on the review sessions panels! if you are interested in 'manual' logins through browser-cookie-copying, please give this a go and let me know which kinds of cookies.txt do and do not work, and how your different site cookie-copy-login tests work in hydrus.

the mappings cache tables now have some new indices that speed up certain kinds of tag search significantly. db update will spend a minute or two generating these indices for existing users

advanced mode users will discover a fun new entry on the help menu

the hyperlinks on the media viewer hover window and a couple of other places are now a custom control that uses any custom browser launch path in options->files and trash

fixed an issue where certain canvas edge-case media clearing events could be caught incorrectly by the manage tags dialog and its subsidiary panels

think I fixed an issue where a client left with a dialog open could sometimes run into trouble later trying to show an idle time maintenance modal popup and give a 'C++ assertion IsRunning()' exception and end up locking the client's ui

manage parsers dialog will now autosort after an add event

the gug panels now normalise example urls

improved some misc service error handling

rewrote some url parsing to stop forcing '+'->' ' in our urls' query texts

fixed some bad error handling for matplotlib import

misc fixes

version 319

started the new convert-query-text-to-gallery-urls object. these objects, which I was thinking of calling 'Searchers', will be called the more specific and practical 'Gallery URL Generators', or GUGs for short

the first version of GUGs is done, and I've written some test ui for advanced users under network->downloader definitions->manage gugs. this ui doesn't save anything yet, but lets you mess around with different values. if we don't think of anything else needed in the next week, I will fix this code for v320 and start filling in defaults

watchers now have a checking slot, much like the recent change to galleries and subs. it safely throttles dozens of threads so they don't rudely hammer your (or the destination server's) CPU if they all happen to want to go at once (like just after your computer wakes up). the option is similarly under options->downloading

moved the new gallery delay/token management code to the better-fit bandwidth manager (it was in domain manager before)

the gallery delay/token code now works per-domain!

moved the gallery delay/token checking code into the network job proper, simplifying a bunch of import-level code and making the text display now appear in the network job control. token consumption now occurs after bandwidth (it is now the last hoop to jump through, which reduces the chance of a pileup in unusual situations) I expect to soon add some kind of 'force-go' action to the cog menu

the network engine will now not permit more than three jobs active per domain, and the overall limit has been raised from ten to fifteen

the media right-click menu now supports copying: all of a files recognised urls; all of a files urls; all selected files' urls of a specific url class; and all selected files urls

reworked and harmonised a bunch of urlparsing and generation code--all urls should now appear as full unicode across the program, generally without %20-type encoding characters unless explicitly entered by the user. character encoding now all happens on the backend in requests

non-url-class-matched urls now have their query parameters alphabetised as part of the normalisation process

all urls in the db will have their query params alphabetised on update, and any file relationships merged to the new/existing normalised url

the manage urls dialog will now normalise newly added urls (but should also still permit the removal of non-normalised urls)

reworked how gallery hits update file import object caches, particularly for subscriptions

fixed an issue in subscriptions gallery logging where the gallery log would always state it had found the max number of files and typically redundantly generate an 'ignored' stub--it should now say something like 'found 7 files - saw 5 previously seen urls, so assuming we caught up' as originally intended

simplified some gallery->file import object creation

galleries now compact until 100 entries (was 25)

watchers now gallery-compact after a successful check

watchers now show the 'just added'/'already watching' status for 15s, up from 5s

network report mode now reports three time--once each for job addition, start, and successful completion

fixed an issue with the new 'max width' popup sizing calculation that was sometimes not fitting for new height requirements correctly

fixed an issue with the new url class next page generation code

fixed an issue where TIOs with data regarding since-deleted services were failing to initialise at the ui level

misc status text cleanup

version 318

downloaders:

extended url classes to support 'next gallery page' generation--a fallback that predicts next gallery page url if the parser cannot provide it (as is often the case with APIs and unreliable next-page-url galleries such as gelbooru)

integrated this new next page generation into new gallery processing pipeline

updated gelbooru, tumblr api and artstation gallery api url classes to support the new next gallery page business

fixed the url class for xbooru, which wasn't recognising gallery urls correctly

wrote new gallery parsers for rule34.paheal and mishimmie (which are both shimmie but have slightly different gallery layout). this should finally solve the 'one paheal gallery url is being parsed into the file list per page' problem

'fixed' the tumblr parser to fetch the 1280px url (tumblr killed the raw url trick this past week)

misc text/status fixes

wrote a gallery parser for tumblr that fetches the actual tumblr post urls and hence uses the new tumblr post parser naturally! (tumblr post urls are now more neatly associated as 'known urls' on files!)

note that as the tumblr downloader now produces different kinds of urls, your tumblr subs will hit your periodic limits the next time they run. they will also re-download any 1280px files that are different to the previously fetched raws due to the above raw change (protip: keep your subscription periodic file limits low)

cut the 'periodic limit' subscription warning popup down to a much simpler statement and moved the accompanying help to a new help button on the edit sub panel

multi-gallery pages now have an 'added' column like multi-watchers

the new 'pause' ? and 'stop' ? characters shown in the multi-downloader pages are now customisable under options->downloading (some users had trouble with the unicode)

the watcher now shows the 'stop character' if checking is 404/DEAD

fixed an issue where the new gallery imports on the same multi-page were all sharing the same identifier for their ephemeral 'downloader instance' bandwidth tracker, which meant they were all sharing the same '100rqs per 5mins' etc... rules

the page and subscription downloader 'gallery page delay' is now program-wide (since both these things can run in mass parallel). let's see how it goes, maybe we'll move it to per-site

subscription queries now auto-compact on sync! this means that surplus old urls will be removed from their caches, keeping the whole object lean and quick to load/save

gallery logs now also compact! they will remove anything older than twice the current death velocity, but always keep the newest 25 regardless of age

.

misc:

the top-right hover window will now always appear--previously, it would only pop up if the client had some ratings services, but this window now handles urls

harmonised 'known urls' view/copy menu to a single code location and added sorted url class labels to entries (which should reduce direct-file-url misclicks)

greatly sped up manage tags dialogs initial calculation of possible actions on a tag alteration event, particularly when the dialog holds 10k+ tags

greatly sped up the second half of this process, when the action choice is applied to the manage tag dialog's current media list

the buttons on the manage tags dialog action popup dialog will now only show a max of 25 rows on their tooltips

some larger->smaller selection events on large pages with many tags should be significantly faster

subscription popups should now 'blank' their network job controls when not working (rather than leaving them on the old job, and without flickery-ly removing the job control completely)

the file cache and gallery log summary controls now have ... ellipsized texts to reduce their max width

fixed an issue where larger 'overriding bandwidth' status wait times would sometimes show instead of legit regular smaller bandwidth wait times

removed a now-superfluous layer of buffering in the thumbnail grid drawing pipeline--it seems to have removed some slight lag/flicker

I may have fixed the issue where a handful of thumbs will sometimes remain undrawn after several fast scrolling events

gave the some-linux-flavours infinitely-expanding popup message problem another pass. there _should_ be an explicit reasonable max width on the thing now

added a 'html5lib not found!' notification to the network->downloaders menu if this library is missing (mostly for users running from source)

help->about now states if lz4 is present

gave 'running from source' help page another pass, including info on running a virtual environment

in file lookup scripts, the full file content now supports string transformations--if this is set to occur, the file will be sent as an addition POST parameter and the content-type set to 'application/x-www-form-urlencoded'. this is a temp fix to see if we can get whatanime.ga working, and may see some more work

if the free space on the db dir partition is < 500MB, the program will not boot

if the free space on the db dir partition is < 1GB, the client will not sync repositories

on boot the client can now attempt to auto-heal a missing local_hashes table. it will give an appropriate error message

misc post-importing-cleanup refactoring

version 317

completely overhauled the tag filter panel:

the tag filter panel now has 'whitelist' and 'blacklist' pages beside the old 'advanced' sub-panel. these new simple pages are much more human friendly for common workflows and provide easy-select checkboxes for namespace classes (which are compiled from all the namespaces your parsers can currently do)

the tag filter rule entering workflow now stops you from creating overcomplicated rulesets: when adding a blacklist rule, it will now only add an explicit entry if it is not already blocked by a higher rule (otherwise it will just discard from whitelist, if there)--and when adding a whitelist rule, it will now only add an explicit entry if it is already blocked by a higher blacklist rule (otherwise it will just discard from blacklist, if there)

tag filters now provide more human-friendly summary statements

misc improvements to tag filter ui logic

the various help texts surrounding the tag filter panel all got passes

the tag filter panel now uses text-and-paste controls for mass-adding of tags

namespace checkboxes have been completely removed from the tag import options panel and various other related places. any existing TIO with checked namespaces will be automatically updated to 'get tags' with an appropriate filter. this is an important step in the rewrite--everything is now handled in the new tag filter panel

simplified and sped up the actual tag filtering code

.

numerous multi-importer improvements:

the gallery and watcher page lists will now ~dynamically~ resize in height based on number of entries, from roughly four columns to twenty four. this relayout code somehow seems to work on all platforms

sped up the 'results loading' step of gallery/watcher highlighting immensely--on a typical list of a couple hundred files, it should now be about 50ms total (before, depending on presentation rules, it could be 0.8-3s)

added an additional db-skipping optimisation for calculating presentation status

watcher and gallery highlights will now filter out trash and completely deleted files (the ones that appear with a dark default 'hydrus' icon) on reloads

added two checkboxes to options->downloading for 'if nothing is highlighted when I add a new X, highlight that new X' for watchers and galleries

adding or removing a query or watcher from the new multi-lists should now be reflected in the list ui instantly, rather than after a <=1s delay

added url classes and parsers for imgur single and multiple urls--thanks to the community for providing some examples

added url class and parser for derpibooru single file pages--again thanks to the community. derpibooru hence now supports basic drag and drop import

fixed an issue where the watcher was often still checking despite 404 status

watchers and galleries use a little less CPU to update some of their ui

added simple subsidiary page parsing support to file import objects (previously, this only worked in the gallery log)

.

gave the thumbnail scrolling code a pass--it is now a bit cleverer about drawing and uses a larger number of smaller 'tile' bmps rather than pages

added an 'EXPERIMENTAL' option to options->gui to change the number of thumbnails each scroll tick scrolls. it defaults to 1.0, but you _should_ be able to set 0.5, 0.37, whatever. please report any bugs!

added a thumbnail debug mode to help see the new thumbnail layout boundaries

.

misc:

the max subscription file limits are now 10,000 for users in advanced mode

the default subs initial/periodic limit is now 100/100 (bumped up from 100/50)

the file import dialog now has a little cog icon to change whether human sort is applied on path addition events (for e.g. if you want to add in some date order from an explorer window)

humansort now sorts case-insensitive

by default, unmatched urls will no longer display in the top-right of the media viewer. see how you like this and let me know if you would like an option to put them back

the speed text on the right-side of the network job control now dynamically resizes to its min size, which gives the text on the left side (where it is often cut off, saying 'overriding bandwidth ...') more space when available

I think I fixed an issue where the popup frame could spam-resize in odd ways (such as growing a pixel wider every update tick)

watchers will no longer include the '* ' highlight prefix in subject-based sort comparisons

in prep for an eventual major code refactoring, the thumbnails' underlying media object now stores a faster db-based numeric file identifier

'duplicate' calls on the new listctrl will now insert the dupes in the current correct sort location, rather than tacking them on the end

drag and drop imports to the new listctrl will also now insert like this

caught up edit subscriptions panel to the finalised common listctrl panel code, including the import/export/duplicate buttons

the multiple checkboxlist selection dialog now sorts by label

converted all old checkboxlist dialogs to the new panel system

massively sped up certain kinds of parsing that were wasting time hitting a cache test way too often

fixed an old hash filtering system

moved to a simpler and more stable way of calculating certain text extents

fixed an issue where the include directory (which has the original source, which isn't a big deal but is nice to have) wasn't being correctly copied into the linux build

the os x .tar.gz build now has the include directory

refactored some client tags code around

misc cleanup

version 316

gallery:

gallery url classes can now be linked to parsers!

if parsers are linked, gallery pagewalk can now work on the new parsing system. gallery import pipeline has been significantly updated to reflect this

gallery import objects are now 'multiple' gallery imports, much like the multi-watcher, with each separate query having its own entry in a list (they also run in parallel!)

the multi-gallery list will show file/gallery pause status in slender columns, and will show a 'stop' character when gallery parsing is done

wrote a 'gallery selector' button and added it to the new multi-gallery page, so you can spawn queries for ~different sites~ on the same import page! it always defaults to 'deviant art' for now, but when the next 'searcher' overhaul step is done, this will be customisable

the new page selector and related 'pages' menu is now simpler--with the new selector, you just select 'gallery'

added 'new_gallery_downloader_page' shortcut action to the 'main gui' set to allow quick opening of this new page type

wrote a 'gallery import panel', which reviews a single gallery import stream, and added it to the multi-gallery page to show the current highlighted query

as all gallery imports now run in parallel and work on the new system, the now almost-useless 'cancel' gallery pagewalk button is now removed

with the wider availability of the new gallery log for file count and error reports, shifted around and smoothed out some gallery status text presentation

improved the auto url_class->parser linking 'try to fill in gaps' logic to work with gallery urls (this was surprisingly complicated)

fixed a misc stupid waste of time in auto url_class->parser linking

many misc updates to gallery pipeline

.

subscriptions:

wrote a new gallery pagewalk pipeline for subscriptions, which still does oldest-to-newest url addition and obeys file limits and so on

numerous subscription pipeline and error handling tweaks and improvements, particularly in regards to the new code

subscriptions now have max initial and periodic file limits of 1000. existing subs with >1000 or infinite will be cut to 1000. there is a help button on the edit sub panel to explain why you should do large syncs with the manual downloader and not subs

the ugly and dangerous-if-you-scroll-in-the-wrong-place gallery selector mismash control in the edit subscription panel is now replaced with the new gallery selector button

fixed an issue where the edit subs panels could sometimes say '48 years ago' (i.e. displaying a literal time delta since 0, 1970) on initial timestamps

juggled some 'periodic limit' reporting logic to skip an unusual false positive that affect hentai foundry subs for now and more in future

.

urls:

the url downloader now accepts gallery urls and will receive drag-and-dropped gallery urls. at the moment, it only parses the one page (i.e. it doesn't start a new 'searching' pagewalk) and sends the parsed links to its file queue

.

watcher:

finished 'watcher panel', which reviews a watcher, and added it to the multi-watcher page to show the current highlighted watcher

the single watcher page is completely removed--it is only the multiple watcher now. all singles will be converted to multiples on update

some single-watcher options (like watchers naming their own page tabs and the [404]-style page name prefixes) are removed

multiple watcher panel now lists file/checking pause status and has separate buttons to control these paused statuses

fixed some misc watcher highlight code--highlighted watchers should correctly publish to the page from the start of session load now

improved some 'repage' logic in how highlighted threads get removed

.

misc:

discovered a scroll-setup parameter that stops janky scroll-to-click-focus behaviour on all the new scrolling panels, thank the LORD

improved some 'can't parse' error handling for post url parsing

reworked how all importers present their network jobs to the ui, including fast response if the switch happens during a job

in prep for searcher switchover where all downloader sources will be harmonised into one system, booru identifiers now present in several ui locations as 'name', not 'booru: name'

updated the danbooru parser to deal with the new 'next page' markup they use

wrote a gelbooru gallery parser that works with 0.2.0 and 0.2.5 gelb sites--an ancillary issue where gelb-related downloaders could sometimes not accurately figure out the magic '42' next-page offset is hence now fixed

wrote an e621 gallery page parser

these sites hence now support single-page gallery drag-and-drop

added url classes for artstation gallery url and its api counterpart, but didn't go further yet--we aren't quite there with api pagewalking just yet

updated deviant art gallery url classes

added an e621 'search initialisation' gallery url class to improve some future drag-and-drop stuff

url normalisation no longer cuts off 'www.'-style prefixes

url comparison is more careful to test 'www.'-style prefixes, so a file import cache should recognise that 'http://www.blah.com/blah' is the same as 'https://blah.com/blah'

did a bunch of refactoring to further split up the bloated ClientImporting.py

fixed some misc downloader layout that may have been hiding some texts previously

some multi-watcher and multi-gallery events like add/pause query should be a bit snappier

in the parsing ui, url and title priorities are now 50 by default

prepped a little subscription unit test code for when searcher object is done

misc downloader layout improvements

misc listctrl refactoring

version 315

got started on the big gallery update, but decided not to pull the trigger just yet. I hope to do it next week, switching the whole thing over to a two-object multi-watcher kind of deal

updated to wxPython 4.0.3 for all platforms

cleaned up some menubar replacement code, and the update to the new wxPython should also fix a "event for a menu without associated window" bug some gtk2 users were seeing on quick menubar changes

manage default tag import options panel now has copy/paste buttons that work on the listctrl

added some 'paste tag import options' safety code to make sure no one accidentally pastes a subscription or something in there, wew

added default checker options for subscriptions to options->downloading

unified how checker options are edited from their button, much like how file and tag import options work. it also has a summary tooltip on the button

the checker options under options->downloading are now these slimmer buttons

in the manual import dialog (which pops up when you drop a folder/files on the client), the files will now be added in 'human friendly' number sorting, so files of the sort 'Favourites - 10.jpg' will sort [10, 11, ..., 99, 100] rather than the purely lexicographic [10, 100, 11, ..., 99]

gave the migrate database dialog a pass--a bunch of misc presentation changes and a general simplification of workflow, now based more on just increase/decrease location weight

a bunch of texts on page management (left-hand) panels that share horizontal space with buttons should now ellipsize ("downlo...") when they get too long for the width instead of drawing in an ugly way over the buttonspace

moved the manage import folders dialog to the new listctrl and added a 'paused' and better 'check period' column

if a user tries to run a 'paused' import folder specifically from the menu, the import folder will now unpause (I will probably remove this old paused variable in the future--it isn't of much use any more)

tightened up some repository reset code that wasn't deleting all service tables and hence recovering from some service id malformation errors correctly

wrote a 'clear orphan tables' db maintenance routine that kills some spare tables some users who have previously deleted/reset repositories may have floating around

fixed an issue with parsing folders after hitting cancel button on the import files pre-dialog

if watchers encounter non-404 network errors during check, they should now just delay checking for four hours (before, they were also pausing checking completely)

if watchers are in 'delay' mode, they'll also not work on files.

file and gallery downloads that hit a 403 (Forbidden) will now present a simpler error status, like they do for 404

the new post downloader will no longer fail if one of the parsed source urls is not a url. the borked string will also not be associated as a url

regular gallery downloads now override bandwidth for the file download step, which is almost always the second half of a pair of post_url/file downloads, just to keep things in sync in edge cases

cleaned up some timestamp generation and 'overriding in x seconds' strings to be more human friendly

improved some serverside file parse error handling to propagate the actual error description up to the client a bit better

fixed typo causing incorrect num_ignored count in file import status button right-click menu

parseexceptions will now present more data about which page and content parser caused the problem. I am not totally happy about how this solution works and may revisit it

the lz4 import error catching is now more broad to catch some odd problem I discovered in new Linux build environment

the moebooru parser now fetches the original png of an image, if available

added a new tumblr parser that also gets post tags--it _shouldn't_ be the default

the new login pipeline now kicks in for the legacy logins--pixiv and hentai foundry--on a per-url basis, so adding pixiv/hf urls to the url downloader will trigger a login even if needed (previously, this was tied to legacy gallery initialisation, which explains some pixiv 'missing' login stuff some users and I were having trouble with)

if the legacy login system fails in the new pipeline, it now sets a flag and won't try again that client boot

the old 'default tag import options' panel is now completely removed from options->importing. please check 'network->downloaders->manage default tag import options' for the new url-based settings

misc fixes

version 314

tag import options can now be set to 'default', meaning 'use whatever the default is at the time of import', which will be an easier way of managing TIOs for many subs that you'd prefer all share the same TIO settings anyway

updated tag import options ui to enable this default setting where appropriate

updated the newer import pipeline to work with 'default'-set tag import options

new downloaders, subscriptions, watchers, and multi-watchers now start with 'default' tag import options

deleted the old default tag import options management code and put some text up on options->importing making notice about the impending shift. a popup message will also say this on update

tag import options buttons now have a right-click menu with copy/paste/default options for quick assignment and duplication!

added 'overwrite tag import options' to manage subscriptions--once you are comfortable with the new 'default' TIO mode, and after some small tests, you might want to switch all your subs over to 'default'

the 'urls' downloader now has a tag import options--it initialises as 'default'

added furry.booru.org to gelbooru 0.2.0 parser

wrote a hentai foundry file page parser

wrote a moebooru file page parser (this works for konachan, yande.re, and sakugabooru)

wrote a shimmie parser (this works for rule34.paheal, rule34hentai, and mishimmie)

wrote a newgrounds parser

integrated the user-created sankaku parser

wrote a tumblr parser that handles photo, photoset, and video posts, auto-converts to 'raw' urls for those post-2012 urls that can handle it, figures out a creator tag (reverting to the reblog root if it is a reblog post!), and cooks you breakfast

(hence all these above sites now support drag and drop!)

rolled out some new tumblr url classes to handle all this.

added sakugabooru url classes

fixed an issue where url classes were not normalising api urls in all cases, meaning some url classes would not api-link correctly in 'manage url class links' panel

fixed an issue with deviant art legacy gallery parser pulling some funky 'creator:' tags

some misc new downloader error handling improvements

the watcher now uses the new gallery object to parse and generate file import objects

the downloaders with gallery logs should now report non-success gallery fetches, along with error tracebacks (this will include some no-worry 404s the legacy downloader sometimes uses to terminate searches)

added image and thumbnail cache timeout time delta buttons to options->speed and memory

added a 'show the D on short file import summaries' checkbox to options->downloading--it defaults to off

the 'I' on short file import summaries is now 'Ig' to clear up 1/I confusion

added 'copy queries' to the edit subscription panel, which lets you copy all the selected queries' search texts to clipboard, newline separated

added a checkbox to options->gui that commands 'last session' only be autosaved during idle time. this is useful if you usually have a huge (200k+ file) session and your client is always on

fixed file import status button right-click, which I messed up somehow last week with the 'retry ignored' add

shook up and collapsed the network menu into neater categories

tightened-up the rarely used pre-parsing conversion panel on the edit page parser panel to just a button with a bit of explaining text

if database errors include the word 'malformed', the client now throws a little extra error text pointing people to the help.txt in the db dir

cleared out some legacy download code

cleared out legacy hard drive import error handling, moving it all to the new file import object

misc refactoring and cleanup

version 313

fleshed out the new gallery log and its constituent log entry objects

added gallery logs to gallery downloaders, subscriptions, url downloaders, simple downloaders and watchers

added very simple gallery log reporting to these downloaders

added first, read-only version of gallery log ui to these downloaders

fleshed out some new gallery/file-object pipeline stuff

wrote a simple danbooru gallery page parser and added it on update. it doesn't do anything yet, but if you are into the new parsing system, please check it out as an example

the url downloader now has a full file import status control with status text

fixed a url count issue on completely fresh gallery downloads that was stopping gallery searches one file (like 199 vs 200) before the file limit

the pixiv downloader now fetches 'type=all' gallery pages, which include specifically manga file pages (as opposed to merely multi-file 'illustrations')

added 'retry ignored' to the file import status button's right-click menu

fixed the deviant art url class and parser to use the new file page format. also added an '(old format)' class to match the old way for legacy purposes (this legacy class also uses an api conversion to connect to the new parser--we'll also figure out a way to convert all these over at the db level en masse later!)

updated some similar deviant art gallery stuff as well

tag import options now has a tag filter to go along with the 'get all tags' checkbox! ('get all tags' is now renamed to 'get all' as a result). this filter lets you make more complicated tag filtering decisions like 'get all tags except "species:" tags'.

the new 'only get tags if they already exist' checkbox now also has a filter, if you want to only apply this test to a subset of tags (like the unwashed mess of unnamespaced tags many boorus and sites provide)

generalised a 'tag filter' button class to make it simpler to edit tag filters across the program, and cleaned up some related status code

fixed a problem with deriving tag import options for specific url classes when that url class was part of an api-url-class chain

if the domain manager cannot now find a url match for a pending download, it now assigns the file post default tag import options to that import

added a new 'duplicates' options page that has a hacky way to edit the weighted scores used to determine which of the pair of files to present file in the duplicates filter

unifed how some file import status generation works, adding a new 'simple status' string to briefly summarise progress in multi-watcher and edit subscriptions columns

cleared out some old redundant status caching in the urls downloader

simplified how almost all timestamp strings are generated

simplified how time delta strings are generated

brushed up some simple common ways to present timestamps as 'human pretty' strings

all places where timestamps would be presented as a mix of '5 days ago' and complete datetime strings will now present as '5 days ago' unless you set the new options->gui 'always show iso' checkbox. going back to simple to clear up confusion in workflow and code. I may revisit this, as turning on ISO mode now spams it all over the place

cleaned up the 'looks like the computer just woke from sleep' check and reduced its grace period to fifteen seconds. foreground daemons (like the subscription daemon) and the network engine will now also obey it

added a 'simulate wake from sleep' debug action to better test the sleep-wake detection code

improved my custom statictext class to auto-wrap text without flickering

used this new autowrapping to improve wrapping and layout of popup message texts

replaced all other st wrapping with this new code

wrote a little helper function to better dedupe lists in future

did a bunch of refactoring to neaten some long common func names

deleted some old unused code

version 312

converted much of the increasingly complicated tag import options to a new sub-object that simplifies a lot of code and makes things easier to serialise and update in future

tag import options now allows you to set whether tags should be applied to new files/already in inbox/already in archive, much like the file import options' 'presentation' checkboxes

tag import options now allows you to set whether tags should be filtered to only those that already have a non-zero current count on that tag service (i.e. only tags that 'already exist')

tag import options now has two 'fetch if already in db' checkboxes--for url and hash matches separately (the hash stuff is advanced, but this new distinction will be of increasing use in the future)

tag import options now applies sibling and parent collapse/expansion before tag filtering, which will improve filtering accuracy (so if you only want creator tags, and a sibling would convert an unnamespaced tag up to a creator, you will now get it)

the old 'all namespaces' checkbox is now removed from some 'defaults' areas, and any default tag import options that had it checked will instead get 'get all' checked as they update

caught up the ui and importer code to deal with these tag import option changes

improved how some 'should download metadata/file' pre-import checking works

moved all complicated 'let's derive some specific tag import options from these defaults' code to the tag import options object itself

wrote some decent unit tests for tag import options

wrote a parser for deviant art. it has source time now, and falls back to the embedded image if the artist has disabled high-res downloading. if it finds a mature content click-through (due to not being logged in), it will now veto and set 'ignored' status (we will revisit this and get high quality nsfw from DA when the login manager works.)

if a check timings object (like for a subscription or watcher) has a 'static' check interval, it will now apply that period to the 'last next check time', so if you set it to check every seven days, starting on Wednesday night, it will now repeatedly check on Wed night, not creep forward a few minutes/hours every time due to applying time to the 'last check completed time'. if you were hit by this, hit 'check now' to reset your next check time to now

the multiple watcher now sorts by status by default, and blank status now sorts below DEAD and the others, so you should get a neat subject-alphabetical sort grouped by interesting-status-first now right from the start

added 'clear all multiwatcher highlights' to 'pages' menu

fixed a typo bug in the new multiple watcher options-setting buttons

added 'retry ignored' buttons to edit subscription/subscriptions panels, so you can retry pixiv manga pages en masse

added 'always show iso time' checkbox to options->gui, which will stop replacing some recent timestamps with '5 minutes ago'

fixed an index-selection issue with compound formulae in the new parsing system

fixed a file progress count status error in subscriptions that was reducing progress rather than increasing range when the post urls created new urls

improved error handling when a file import object's index can't be figured out in the file import list

to clear up confusion, the crash recovery dialog now puts the name of the default session it would like to try loading on its ok button

the new listctrl class will now always sort strings in a case-insensitive way

wrote a simple 'fetch a url' debug routine for the help->debug menu that will help better diagnose various parse and login issues in future

fixed an issue where the autocomplete dropdown float window could sometimes get stuck in 'show float' mode when it spawned a new window while having focus (usually due to activating/right-clicking a tag in the list and hitting 'show in new page'). any other instances of the dropdown getting stuck on should now also be fixable/fixed with a simple page change

improved how some checkbox menu data is handled

started work on a gallery log, which will record and action gallery urls in the new system much like the file import status area

significant refactoring of file import objects--there are now 'file seeds' and 'gallery seeds'

added an interesting new 'alterate' duplicate example to duplicates help

brushed off and added some more examples to duplicates help, thanks to users for the contributions

misc refactoring

version 311

wrote a new parser that muddles its way through pixiv's new dynamic javascript layout. it seems to get everything working again. it gets tags in kanji, although the unnamespaced pixiv tags remain low quality, and you may wish to just not parse them at all anyway

fixed some misc parser text handling, unicode conversion etc...

the new pixiv parser has a 'page' tag stub that should inform tag import options in the old downloader

the multiple watcher now remembers the highlighted watcher through a session restart

the multiple watcher now shows the highlighted watcher's url up top

the multiple watcher now has checker, file import, and tag import options, which it will assign to all new watchers it creates

the multiple watcher now has a 'set options to watchers' button that will force-set the current options to all the selected watchers

the multiple watcher now has an 'added' column with watcher creation time listed. storing this creation time is new, so any existing watchers will get a new creation time of their next load time, but it is remembered henceforth. the listctrl here is now pretty crushed for width, so maybe we'll rejigger some stuff here

watchers added to a multiple watcher will now have a status of 'just added' for five seconds

watchers that are added to a multiple watcher that is already watching them will now have the status of 'already watching' for five seconds

the multiple watcher list now has a much taller minimum height--layout here is another work in progress

fixed the inkbunny parser (and a related tweak to the inkbunny url class)--it now uses the new 'multiple-file-per-post' import object generation to actually walk through the pages of the mini-gallery (which for inkbunny have -p2- suffixes on the url) to fetch only the correct files and url-associate them neatly

tag import options now has a 'get all tags' checkbox, which can override the normal namespace checkboxes. it gets all tags, even those with namespaces not listed, which happens for several reasons in the new download system. (eventually, the namespace list may be replaced with a slightly different system)

watcher tag import options no longer list 'filename' under their namespace checkboxes--they just have this 'get all tags', which works for everything (so watching yiff.party pages should now get tags)

simplified and sped up similar files search at the db level

sped up some ratings search code

generalised some common file search optimisations, meaning they now apply in more situations and can take advantage of some other speed-ups:

similar files system predicate is now faster

inclusive ratings searches are now faster

duplicate relationship count searches with non-zero-inclusive count are now faster

removed some clumsy old ratings search optimisation code

exporting serialised objects as pngs is a bit easier--now, it displays current export path better, will remember the last export location used, and for single png exports will pre-fill the filename and 'title' value with a reasonable default

the content parser, page parser, and url class listctrls now accept serialised png files when drag and dropped!

the simple downloader should recover and continue better from malformed urls during a page parse

the url downloader should now recover better from various situations where it cannot not derive some tag import options (including urls with a 'file' url class, such as 4ch/8ch direct file links)

parse test results will now state the priority value of urls

gave the 'updating' section of help a pass and wrote a little more on how to do a big-version-gap update

when a new multi-file import object inserts its child file import objects while being looked at in the ui, the listctrl should now correctly refresh the displayed indices

subscriptions will now wait up to 90s for bandwidth (was 30s before, I think) before quitting, which should avoid a few more early-quit events

cleaned up some server decompression bomb testing

users with admin-level accounts can now upload decompression bombs to file repositories, better options on this will be avaliable in future

the manage urls dialog will now OK on the same 'manage_file_urls' shortcut action that can open it (like manage tags and ratings already do)

fixed the string converter for new file lookup parsing scripts

started work on some in-the-background mass file reparsing, but I want to get some nicer ui going before I pull the trigger on any of it

file reparsing now repopulates the table for md5, sha1, and sha512 hashes if they are missing

improved some ffmpeg error parsing

moved from basic list to a pop-faster collections.deque for importable path parsing and duplicate search branch regen

added a BUGFIX option to options->gui that forces minimum width for popup messages in the continuing attempt to deal with some funny fit/layout calculation in certain Linux WMs

fixed how some 'unrepairable db' error messages are displayed in Linux systems

cleaned up a ton of old tuple-stripping code from the db

updated to new sqlite for windows build

misc improvements

version 310

updated the inkbunny file page url class to acknowledge that inbunny pages can have multiple files

updated the inkbunny file page parser to handle multiple file urls (although they may be out of order and possibly sometimes include the artist profile image--this was not super easy)

added a parser for twitter tweets (only images supported atm, but it can handle multiple!) (hence tweet drag and drop now works!)

updated the artstation file page url class to redirect to a new api url class

wrote an artstation file page parser that also handles multiple file urls

updated/added pixiv file page, manga page, and mange_big url classes

updated pixiv file page parsers to be ok with manga links

wrote parsers for pixiv manga and manga_big pages to fetch manga files (with page tags)!

file import objects can now create semi-duplicate children for multi-file post urls and insert them just after themselves in the file import queue.

file import objects can now receive and remember referral urls. this referral url is associated with the file if appropriate. the watcher and simple downloader now uses this in addition to the multi-file post system

jumbled around some parameters and merged the two new file import url commands (import 'file' vs import 'post') into one single simple 'work on this url, thanks' call that is now used across the program

the parsing system's 'content parser' no longer fetches file urls and post urls, but 'download urls' and 'source urls'. this helps some pipeline logic and also lets post urls be download urls

when file import objects parse post urls as the urls to download, it now creates 1-n new import objects, just like if multiple file urls.

improved some file import object file association code

the new parsing system will de-dupe parsed urls

refactored the 'seed' code, which handles all basic file import objects, to the new ClientImportSeeds.py

added a new string transformation type, 'integer addition', for shifting page number tags up and down

fixed thumbnail generation for some videos that failed to do the new x%-in generation--it reverts more reliably just to the old frame 0 method

file reparsing popup now has a stop button

fixed an issue where extremely thin or wide (ratio > 200:1) images would not generate a full-size or resized thumbnail

the file reparsing/re-thumbnailing now reports errors better (including with full path) and does not abandon the larger job as it works

misc thumbnail generation code improvements

improved some thumbnail and file regeneration/moving code when the existing file has read-only status

the multiple watcher now has a 'check now' button

added a checkbox to options->gui that will put new notebook page tabs on the left

for all file download network jobs working in the new download system, the file import options for min size, max size, and max size (gifs) are now applied _during the download_! if the server tells the client the exact file size in the response headers, it will test max and min size before the content is actually downloaded--otherwise, it will test the max size as it downloads. if the server clearly says the file is a gif, the max gif size rules will also be tested in the same way

cleaned up some bandwidth announcement code--now, if bandwidth is due in less time than override time, that will now correctly be the status text

the bandwidth status no longer says 'in in' typo

fixed up some tag repair code from last week

the 'print garbage' debug function now dumps a whole bunch more data to the log

the thumbnail cache should now be a bit more stoic about missing repository thumbnails--it should now just present the hydrus default backup without error popup spam

the repository thumbnail sync will now get as thumbs in blocks as high as 10k at a time, rather than the old 100

hydrus network requests no longer generate web domain network contexts (and so won't have a default one-request-per-second bandwidth limit and should stream through thumbnails a bit faster)

hydrus network services are now willing to wait longer for bandwidth, so big thumbnail queues should keep working even if other bottlenecks pause them for a bit

hydrus network services will no longer sometimes have double-sync popups if synced from the advanced 'sync now' button in review services

changed the default global 'stop-accidents' bandwidth rule of 120rqs per minute to 512MB per minute. this only affects new users, but users trying to sync to large file repos might like to make a similar change manually

doing giant full file delete (i.e. purge from trash) jobs should now be a bit gentler on the gui

improved how the client deletes paths, clarifying in the code when and when not to allow recycle (usually disabled for thumb disposal)

switched the hacky text widgets on the popup system to a newer object. seems to still render ok, so lets see if it fixes some unusual layout issues some users have seen

if the temp folder cannot be created on boot, the client will continue anyway

fixed some url-domain text handling in db storage that was also breaking v309 update for some users

fixed some additional domain generation error handling at the db level

the list of url classes in the system:url panel is now the list of all url classes that are considered associable (before, it was file and post urls)

if a url class now api-links to itself or otherwise forms a loop with n other api url classes, the client will now throw an error (rather than lock up in an infinite loop!)

in the parsing ui, tag parse test results are now cleaned before being displayed

fixed misc url matching error reporting bug

when consulting the current file limit, the gallery page downloader will now try, when it has that number, to consult the total number of urls found it the current search (old behaviour is to only consult the number of _new_ urls, which lead to some bad edge-case workflows)

misc refactoring

version 309

wrote a fix for the tumblr GDPR issue under _network->DEBUG: misc->do tumblr GDPR click-through_. you will also get a popup about this on update

the tumblr downloader will try to detect the GDPR problem and present a similar popup guiding you to the GDPR click-through solution

the client and server now generate video (but not gif yet) thumbnails 35% in by default. the client can now change this percentage value under options->media. this was highly requested and was being put off for a longer rewrite, but I figured out a simple way to hack it in. please let me know if you get failures

on adding a parent, all files with the child tag will now also get all applicable grandparents (with no limit on recursive generations and dealing with accidental loops)

on adding a sibling, all files with any of the siblings will now also get all applicable parents and grandparents for the whole group. a maintenance call to retroactively fill in the sibling/parent gaps that are now filled will also come soon

this logic still does not apply in cross-service situations, which _will_ likely have to wait for a big data/gui overhaul and us figuring out what we actually want here

added a simple pause/play button to the multiple watcher

if the multiple watcher is set to catch watchable url drag and drop events and the current page is a multiple watcher, this current page will catch those new urls (as opposed to the _leftmost_ multiple watcher)

improved some thread unpause logic which was failing to lock pause during 404 status

the multiple watcher should now ignore case when it sorts by subject

added url class and file page parser for inkbunny (so this site is now supported in drag and drop!). it fetches creator tag, some artist-made unnamespaced tags, source time, and md5

added file page parser for gelbooru 0.2.0, which by default works for rule34.xxx, tbib, xbooru but certainly should work for a bunch of others. it fetches source time and source url

html formula parsing rules can now additionally test the tag 'string' using a standard StringMatch object. this greatly helps to parse otherwise indistinguishable 'a' tags that have string 'Original image' and so on

the 'have I seen this url's file before?' pre-import test is now much more strict and will cause fewer accidental false-positive 'already in db'/'deleted' results:

the url pre-import test now does not trust source urls if they do not have a url class

the url pre-import test now no longer trusts urls that are supposed to only be mapped to one file but are actually mapped to multiple

this url pre-import test now treats url-classless original post urls and intended file urls with a special level of trust

urls are now stored in the db in a more powerful and in-future easily searchable way--your db will take a moment to convert to the new format on update

did some prep work for multi-file post urls (like pixiv manga) but did not have time to finish it

the filename tagging options panel (in the 'add tags based on filename' of file import dialog and import folder dialog) now updates its tags/list 0.5s after the last change event, which means typing on a giant list will not cause megalag

improved stability of some client-screen coordinate conversion

misc bmp handling stability improvements

improved some parsing ui stability when example data gets set after the dialog is closed

improved some misc dialog close stability

converted all but one final ui update timer to the new job scheduling system

there are still problems with linux stability--I will continue to work on it

an ugly (but basically harmless) shutdown exception sometimes caused by Animations being a bit slow on deleting their underlying bmps _should_ be fixed

the export files dialog now generates its paths in sort order, meaning (1), (2) de-dupe filename suffixes should now be generated nicely in order

the network domain manager should now always chase API URL links to get the right parser

made some 'the db is broke, let's try to fix it' tag recovery code more forgiving

misc improvements to some media indexing backend, which may fix some unusual session ghost files

fixed the 'sure it is ok to close this importing page' dialog to also veto on a 'cancel' event, rather than just a 'no'

added a guide to database_migration.html on how to move the db from just an HDD to straddle both an SSD and HDD.

cleaned up the help->debug menu a bunch

added run fast/slow memory maintenance calls to help->debug->data actions

misc cleanup

version 308

the multiple watcher will now discard new urls if it is already watching them

the multiple watcher will list x/y progress as just 'x' if x==y (making it easier to scan the list)

the multiple watcher now lists a couple of 'total' summary lines on its ui--the top lists total number of watchers and queue progress, the bottom lists the usual '23 successful, 3 deleted' line, but summed for all watchers

the multiple watcher will now warn you if you try to remove the highlit, alive or un-caught-up watchers

the multiple watcher will now resort if the thread subject (or rather, any data in the current sort column) changes (which usually happens right after it is added, when you see it change from from 'unknown subject' to 'mlp is kino')

fixed an issue where multiple watchers were not unscheduling down their update job correctly on page close

the booru selector in the edit subscription panel should now be in the tab traversal order for keyboard/automated focusing tasks

the boorus in that selector are now alphabetised

tag import options namespaces are now alphabetised

removed/renamed pretty much all references to 'thread' in the watcher code and ui presentation, since it can now do a bunch of other stuff. it is now just the 'watcher' and the 'multiple watcher'

deleted a bunch of old static thread watcher and page of images code from the old downloading system

added an experimental 'compact' button to advanced mode users' manage subscriptions panels. this removes urls from the selected subscriptions' caches that are no longer useful, keeping their load/save snappy. this is still in testing--be careful with it!

the hydrus splash screen now has a bare frame caption and will appear in the taskbar--which helps with some alt-tab and 'where the hell did it go?' stuff if you need to enter a password

wrote five 'reasonable defaults' buttons for the 'check timings' options panel for quick entry for different thread/subscription scenarios.

added a checkbox to this panel that will swap the reactive options with a simpler single checkbox

also clarified/fleshed out the help button on this panel

fixed an important source of program instability related to page alive/dead status checking that was inadvertantly talking subtly to the main gui frame even on non ui threads

improved how some 'page is closed but not destroyed' test logic for pages inside a closed-but-not-destroyed notebook

fixed another small place where the db was talking to the main gui object about status bar updates in a potentially unstable way

fixed another small place where the foreground daemons were talking to the main gui frame in a trivial but potentially unstable way

played around with some taglist sizer and layout settings

the gallery and simple download pages are now a little shorter--the pause and cancel buttons are now just to the right of the status texts, rather than on their own row beneath the network job controls.

the various bandwidth-overriding network jobs in the download system--like gallery page downloading--now wait 30s before overriding their bandwidth. hence these jobs will now obey the usual bandwidth rules up to a point

the simple downloader also obeys the usual bandwidth rules for 30s but no longer has a static wait, so it can run much faster in certain situations

network jobs that will override bandwidth in the future will now report that countdown in their status texts

fixed a bug in the old booru code that meant some boorus were superfluously requesting the 0th indexed page of a gallery more frequently than needed in order to reestablish a 'page size' cache. this value is now cached globally and will be replaced by a completely different system in the new gallery downloader

added a decent tooltip to the 'gallery fixed delay' widgets in the options->downloading panel

the autocomplete input should clear itself after a 'broadcast' event a bit quicker and stop some dupe inputs in certain edge cases

the tumblr url class now recognises that tumblr posts can have multiple files, which helps some source url lookup logic

added a url class for artstation file pages

the primary file import url (the one listed in the file import list) will now correctly not associate with the resulting file if its url class is so set

all the import objects now have much lower idle CPU time and thread needs and start in slightly offset times, smoothing out the thread count spikes

all the import objects will now respond quickly to changes to the underlying file import cache (like right-click->try again events)

the new job scheduling system now uses two queues--fast and slow, in order to reduce some resort/insert overhead

a couple more improvements to the new job scheduling system to smooth out spikes

if the temporary path override does not exist, the client will now compain with spammy popup messages and fall back to the default

if the temporary path override does not exist or is not writeable-to on options dialog ok, a veto exception will be raised

refactored the watcher and multiple watcher to their own file, ClientImportWatchers

misc fixes

version 307

wrote a gelbooru 0.2.5 (which matches gelbooru itself) parser in the new system. it now has some more redundancy and produces md5 hash and source urls

fixed the e621 parser for flash files

manage tags is now a notebook rather than a listbook!

a problem where OS X was displaying the wrong label in the manage tags listbook is now fixed (as it no longer uses the buggy old listbook)

improved autocomplete focus setting in OS X in general

moved all paged importer loops to the new job scheduling system, which will massively cut down on idle time thread count (and some idle CPU usage) on clients with a bunch of import pages open

the main thread pool (which the job scheduling system uses) can now temporarily grow much larger (200 threads) when it gets hammered

recalibrated some thread watcher code to deal with being embedded in a larger object better (and sharing a page with other watcher importers)

finished the first version of a multiple watcher. it is ugly and only for advanced users for now--please check release post for more information

added 'use the multiple watcher on DnD events' to options->downloading, which will send thread url drag and drops straight to a new/existing multiple watcher!

gui session load now 'shows' the first page but loads everything to the right in the background, which saves some CPU and memory for large sessions. this means it now starts focused on the leftmost page rather than the right--see if you like it or not

'clear and load session' now fully deletes old pages and takes a little break between the clear and load step to make sure old large sessions are deleted tidily before loading the new stuff

cleaned up the new leaner popup message display code to react better to (and re-layout) sub-changes in its popups (so now when a subscription popup spawns a new progress gauge, it _should_ be positioned in the right place immediately)

if an import folder/subscription publishes files to a page while the main gui is minimised (and the action would result in a new page creation), the page add will be delayed until the gui is no longer minimised (new page layout in this case was broken due to minimised parent frame)

the edit subscription query panel will now spawn with the query text input focused

added a patch to ensure new booru import pages get query input focus after a second if the focus attempt on init failed (this will be fixed better in the new downloader system)

restored the 'clear deleted files record' button to the 'combined local files' service on _review services_, which was previously hidden during a significant service rewrite.

if critical master tables are missing on boot, the client will warn and provide info and then abandon the boot

if cache similar files tables are missing on boot, the client will warn and provide info and offer to recreate them empty and try to boot the client anyway (this replaces some old repair code that wasn't always kicking in at a good time)

if cache autocomplete tables are missing on boot, the client will warn and provide info and offer to recreate and repopulate them

if mappings tables are missing on boot, the client will warn and provide info and offer to recreate them empty and try to boot the client anyway

the emergency 'repair missing file locations' panel now runs in a more stable way

this emergency dialog will now also recognise 'I fixed all the file paths but cannot do the thumbnail paths' situations and will present the same 'ok, make sure you run regen thumbs after boot' dialog as if all the missing paths were thumbnail related

you can now turn off x/y page name import progress under options->gui

fixed an issue where tags that begin with ':' like ':p' were not getting through the new importing system

fixed an issue where tags that begin with ':' like ':p' could sometimes get prepended extra escape characters and end up looking like ':::p'

fixed sibling tag searching (e.g. if a->b exists and you search for b, results with a but not b should also appear), which the recent tag search optimisation accidentally disabled

whole bunch of refactoring and cleanup of ClientData and ClientGUICommon

cleaned up some spammy splash text status setting that was flooding some debug info with useless garbage

gave getting_started_tags.html's tag repo section and access_keys.html a pass, updating the ancient screenshot and linking advanced users to the latest QuickSync location

some misc help updates

misc autocomplete logic improvements

misc ui display fixes

misc wx destroy code stability improvement

updated ffmpeg for windows builds to the new 4.0 release

updated sqlite for windows builds

version 306

the file import status list now has 'open selected import files in a new page', which should show up where it is possible. this is a bit prototype and ugly--it'll show _all_ files, including in-trash and permanently deleted (which will show up with the hydrus thumbnail)

the file import status list now prefixes the already in db/deleted notes with 'url' or the hash type that lead to the recognition

these redundant/deleted notes now also propagate up from 'during import' recognition phase as well

the 'delete seeds of type x' entries on the file import status button's right-click menu are now split into three smaller individual tyes and are more explicit about exactly which status types they will remove

like import folders, subscriptions can now optionally publish their files to pages as well as popup buttons. also, subscriptions can optionally publish their files separately for each query instead of all merged together

sped up multiple tag queries significantly

sped up simple (file size, mime, etc...) system predicate queries that also include a tag/namespace/wildcard predicate significantly

added a pixiv parser that pulls the japanese tags to the defaults--users can switch to this if they prefer under network->manage url class links

fixed the 4chan parser to get part of comment as backup subject/page title

removed the 'newgrounds' entry from the normal gallery page creation ui, as the basic gallery parser no longer works due to a dynamic loading change on their end. I hope to have it back with the new gallery parsing system I will soon be writing

the edit url classes panel now has a little text box to put in example urls and see which class, if any, that they match to

improved layout of edit url class links panel

all url types are now displayable in the media viewer--only post url classes are default on

the new (x/y) import page page_name progress count is now updated on all alterations to this value (previously, this was not updating when a user interacted with the import queue, only when the natural downloader loop cycled)

added 'can produce multiple files' option to post url url classes, which informs client url-checking logic whether the url can be relied upon for 'already in db/deleted' calculations

the pixiv file page url class now has 'can produce multiple files' checked, meaning some bad pixiv url association logic due to other sites referencing it as a source url is now fixed

added a 'twitter tweet' url class, which is also a 'can produce multiple files' post url

added a 'sync known urls?' action choice to the duplicate merge options panel, which governs whether urls should be copied from worse to better or in both directions

gave the edit duplicate merge options panel a layout pass

the edit duplicate merge options panel will now disable pointless/over-complicated choices on non-custom actions, let me know if this is a pain for your workflow

added a 'manual' web browser path override to the 'files and trash' options panel, which fixes the new share->open->in web browser option for Windows and also fixes some #anchor link propagation

consolidated all URL/Path web browser launching code to one location

'open in web browser' is now available for non-advanced_mode users and the 'open' submenu of the share menu is available in the preview window and the media viewer

fixed a bug that was causing import folders to publish incorrect file identifiers, which was poisoning popup buttons and import page destinations

gui sessions that fail to load a page will recover and continue to attempt loading the rest of their pages. some popups detailing the page's serialised data and error will be presented

gui sessions that fail to save a page will recover and continue to attempt saving the rest of their pages. some popups detailsing the page's rough info and error will be presented

the core controller inside all media pages will now present itself in a more beautiful way when asked to dump itself to a log (which should beautify the above save error a bit)

wrote a subsidiary database->check->just repo update files that tests integrity of only repository update files

fixed an issue where default tag import options were sometimes not being saved from the new dialog in the networking menu

wrote a couple of layers of bad tag protection to help the new downloader deal with some occasional bad output from the old downloader

network jobs can now reattempt connection attempts up to three times on POST requests (if you ever got inexplicable immediate 'could not connect' errors on repository uploads, this should now be fixed)

replaced some archaic misc old import code with the new system, cleaning up a bunch of stuff and making space for further refactoring along the way

fixed tags blacklist not being inherited in the old (through options dialog) system

improved some invalid domain error handling

fixed an animation update issue that would pause naturally updating controls on non-main-gui  frames when there were no regular media pages open on the main gui

added a BUGFIX option to 'files and trash' option page to override the default temp path for almost all client temp path requests

the minimum value for the 'vacuum period' in maintenance and processing options is now 28 days. the control also has a little explanatory tooltip

the 'try to auto-link url classes and parsers' function now always preferences parsers alphabetically

fixed a typo in the string transformations prettyfication code that incorrectly summarised 'take the last x characters' as the opposite

misc fix to file hash generation and status checking code

the 'export tags to .txt files' checkbox on the export files panel will no longer bother you with a dialog as you uncheck it

wrote some code to make it easier and more fool-proof to update the domain manager with new url classes and parsers on my end

improved some popup message manager ok-to-alter-ui logic when the main ui is minimised and so on

fixed some potential crash conditions (affecting linux mostly, seems like) in the service credential testing and access key fetching ui code

fixed a bug when 'stopping' a gallery parse during a long error pause (like when it holds on '404')

sped up some old set intersection code

some import file presentation refactoring

some url content application pipeline cleanup

misc cleanup

version 305

fixed the pixiv url class, which was unintentionally removing a parameter

wrote a pixiv parser in the new system, fixing a whole bunch of tag parsing along the way, and also parses 'source time'! by default, pixiv now fetches the translated/romaji versions of tags

finished a safebooru parser that also handles source time and source urls

finished an e621 parser that also handles source time and source urls and hash!

wrote a danbooru parser that also handles source time and source urls and hash!

as a result, danbooru, safebooru, e621, and pixiv post urls are now drag-and-droppable onto the client!

finished up a full yiff.party watcher from another contribution by @cuddlebear on the discord, including url classes and a full parser, meaning yiff.party artist urls are now droppable onto the client and will spawn thread watchers (I expect to add some kind of subscription support for watchers in the future). inline links are supported, and there is source time and limited filename: and hash parsing

fixed some thread watcher tag association problems in the new system

when pages put an (x) number after their name for number of files, they will now also put an (x/y) import total (if appropriate and not complete) as well. this also sums up through page of pages!

if a call to close a page of pages or the application would present more than one page's 'I am still importing' complaint, all the complaints are now summarised in a single yes/no dialog

url downloader pages now run a 'are you sure you want to close this page' when their import queues are unfinished and unpaused

if the subscriptions for 'manage subscriptions' take more than a second to load, a popup will come up with load progress. the popup is cancellable

added a prototype 'open in web browser' to the thumbnail right-click share menu. it will only appear in windows if you are in advanced mode, as atm it mostly just launches the file in the default program, not browser. I will keep working on this

harmonised more old download code into a single location in the new system

created a neater network job factory system for generalised network requests at the import job level

created a neater presentation context factory system for generalised and reliable set/clear network job ui presentation at the import job level

moved the new downloader simple-file-download-and-import to the new file object and harmonised all downloader code to call this single location where possible

did the same thing with download-post-and-then-fetch-tags-and-file job and added hooks for in the subscription and gallery downloader loops (where a parser match for the url is found)

the simple downloader and urls downloader now use 'downloader instance' network jobs, so they obey a couple more bandwidth rules

harmonised how imported media is then presented to pages as thumbnails through the new main import object

the new post downloader sets up referral urls for the file download (which are needed for pixiv and anything else picky) automatically

improved file download/import error reporting a little

entering an invalid regex phrase in the stringmatch panel (as happens all the time as you type it) will now present the error in the status area rather than spamming popups

fixed a bug in the new parsing gui that was prohibiting editing a date decode string transformation

fixed enabling of additional date decode controls in the string transformations edit panel

added a hyperlink to date decoding controls that links to python date decoding explainer

if a source time in the new parsing system suggests a time in the future, it will now clip to 30s ago

misc downloader refactoring and cleanup

fixed an issue where new file lookup scripts were initialising with bad string transformation rows and breaking the whole dialog in subsequent calls, fugg

hid the 'find similar files' menu entry for images that have duration (gifs and apngs), which are not yet supported

added 'flip_debug_force_idle_mode_do_not_set_this' to main_gui shortcut set. only set it if you are an advanced user and prepared for the potential consequences

silenced a problem with newgrounds gallery parser--will fix it properly next week

fixed some old busted unit test code

rejiggered some thumb dupe menu entry layout

version 304

renamed the new 'tagcensor' object to 'tagfilter' (since it will end up doing a bunch of non-censoring jobs) and refactored it into clienttags

attached a tag filter object to all tag import options to act as a tag blacklist. all tags that go through the import pipeline (except for a couple of old legacy instances) are now checked against the blacklist, and if a bad tag is found, the file vetoes! tag import options has some new ui to handle this and background code to deal with inheritance from defaults and so on

new file import urls that have url classes, no matter their source, are now normalised!

all new file import urls are now tested against both the original and normalised version of the url, so even though previously parsed urls remain un-normalised, new urls that are pre-normalised the same will not count as new! -fingers crossed-

on update, the db will get normalised copies of all existing urls. this means many files will now have two versions of its urls--some ui to collapse everything down to only the normalised version (after some human eyes have passed in front of this big change) will come in the coming weeks

some sites where normalisation is a consistent problem for later redownloads (like e621, which appends 'preview' tags to the post url) _should_ now be caught reliably!

the 'allow subdomains' on edit url class panel is now named 'match subdomains' and has a tooltip to better explain how it works

'keep subdomains' is now 'keep matched subdomains' and has a tooltip as well

the 'keep matched subdomains' enabled behaviour (and some normalisation calculation) is now additionally governed by the 'associate url with files' value and api url conversion info rather than just 'match subdomains' and raw url type

fixed an issue that was stopping the 'associate url with files' option sticking in edit url class panel

edit url matches now resorts after an add or edit action

all listctrls with a wrapper panel now resort after an import from clipboard, png, or defaults call

url matches now match against www*. versions of their domain regardless of 'match subdomains' settings

updated xbooru url classes to prefer https

the manage url class links panel now has a 'clear' button to clear a url_class->parser link

introduced three new simple downloader parsers for yiff.party, thanks to @cuddlebear on discord for the submission

the old 'uninteresting mime' status has been expanded to a wider 'vetoed' status to represent all file imports that are abandoned without a particular error (e.g. tag blacklist, wrong filesize or resolution)

the import system now reports the total of 'num vetoed' as 'num ignored' in its summary statements

it now also reports 'num skipped'

the 'num successful' and 'num already in db' are now folded more neatly together in import cache summary statements

file downloads that are cancelled will now set a 'veto' state rather than a 'skip' state

improved file import exception handling across the board

improved how single-file-result parsing vetoes propagate up to the file import status cache

404 network errors will now provide a 'veto' status rather than an 'error'

vetoes will not count as errors when deciding whether a subscription should be abandoned early (so a bunch of decomp bombs or 404s will no longer stutter a subscription!)

misc fixes and improvements to the new download stuff

wrote a new parsing cache that saves a lot of work in the new parsing system

improved the 'is this url known?' test to better deal with situations where all the given urls are galleries or unrecognised--a better aggregate of file status is formed, and 'already in db'/'deleted' statuses will apply if there is no evidence otherwise (the dev got the new logic for this from a legit nightmare about urls downloading over and over, so let's hope it works out)

the 'is this url known?' logic also recovers from 1->n url->hash relationships where it does not expect them, trying to find 'already in db' hashes over 'deleted' ones

to clear up some ambiguity, galleries or subscriptions now give a different 'checking in x seconds' status when waiting on the first page of a query

the 'noneablebytescontrol', as seen in edit file import options, will now correctly disable/enable its bytes sub-control when it is none'ed

a persistent issue with the new network engine sometimes failing to correctly error after certain broken connections (the computer going to sleep mid-download was a common cause here) should now be recovered from and the connection naturally reattempted

added three new shortcuts to the 'main_gui' shortcut set that allow for opening a new 'urls', 'simple', or 'thread watcher' downloader page

added two more shortcuts to 'main_gui' for new 'page of pages' and 'duplicate filter page'

moved some old 'new page' menu code to the new application command system

added numerous 'duplicates' shortcuts to the 'media' shortcut set that will work on selections of thumbnails

the thumbnail duplicates menu actions now go through the new application command system

fixed an issue where the current tag parents caches was not refreshing when notified

inputting a short invalid syntactic input on a 'read' tag autocomplete such as '-' will now clear the system predicates list--system preds should now only show on a completely empty input

fixed an issue where certain combinations of 'remove a tag, then re-add it' nullipotent actions in a single manage tags dialog transaction were not applying reliably (sometimes, the subsequent mirror action was not occuring due to a processing re-order optimisation at the db level)

made some animation code a little safer and quieter as a test for some users who were getting blitzed with some deadwindow error spam in certain situations--let's see if this changes anything

replaced all the em dashes in the help with double hyphens as github pages was rendering them wrong

added CrystalDiskInfo recommendation to 'help my db is broke.txt'

misc cleanup

version 303

file post url classes can now be linked to parsers!

the 'raw url' downloader is now just the 'url' downloader. if a dropped url is matched as a 'file post' url and links to a parser, it will download it with that parser and default tag import options

url drag and drop now recognises file post urls that have linked parsers and will send them on to a url import page, just like 'raw' urls! they will use the new parsing system to parse tags and known urls and all that

wrote a way for the new download system to store and edit default tag import options--this can now be found under network->manage default tag import options, although it does not do much yet. eventually the old options will be transferred here, and the different downloaders will consult it more

refactored and cleaned up how some default tag import option calculation is done

the fixed 5 second wait time between gallery page fetches is now editable for downloader pages and subscriptions under options->downloading. subscriptions default remains 5s, gallery default is now 15s

the gallery downloader will present its 'x urls found so far' string in a more uniform and reactive way. it will also count down until the next page fetch

subscriptions will now similarly react quicker while the gallery parse step is ongoing, and it will count down to the next page fetch

the manage subscriptions dialog will now note in BIG RED TEXT if subscriptions are currently globally paused

subscriptions now consume less parallel-timeslot overhead, meaning they sync much faster when they have no work to do (which is most of the time)

separating subscriptions is now more sophisticated--if you wish, you can now choose just to separate a subset of the large subs's queries, and if you do so, you can decide whether to merge the extracted subs into a new larger sub or just have them as individuals. also, you will be asked for name/base_name for what to name the new subs

since separating subs is more complicated, the button is only enabled when a single large sub is selected. please do your separating one sub at a time!

merging subs will now ask if you want to rename the merged subs as it goes

refactored a giant heap of duplicate import code into single locations in the new file import object--things like applying tags and generating pre-import status are all now done in one place. a number of weird behaviours (like not applying known url associations in certain circumstances) should now be a lot better and unified

the way the db and other import testers handle whether a pending import is new or unknown is simplified

optimised the way the importers figure out whether to display a new import

the 'has the client seen this url before?' test used to figure out whether to download a file now explicitly ignores anything it isn't certain is a single-file File URL or Post URL (according to current url classes). associating 'gallery' known urls is no longer catastrophic to this system

the simple downloader now associates the entered url with the files subsequently imported

file import objects working in the new parsing system are aware of the new 'should this url be associated' option and will be given gallery urls to hold on to as appropriate

url classes now have an explicit option as to whether they should be associated with files imported from them, which is not appropriate for dynamic CDN File URLs but is appropriate for multi-image tweet links (which will likely be Gallery URLs in the new system), for instance

general importer code cleanup across the board

threw together some early ui to show current jobs in the network engine under network->review network jobs

popup messages will now try to set a max width of roughly 56 characters wide rather than 400 pixels. this value is editable under options->gui

updated the listctrl in the export files frame and the way this frame generates and stores export filenames. let's see if linux users who had crashes with this have better luck

refactored stuff out of the bloated ClientImporting file into seperate new files

refactored client network bandwidth and session code out to their own files as well

refactored client network objects to its own file

cleaned up some open externally debug code linked to callto report mode

a common grid layout used across the program will copy control tooltips to the string on the left

removed the 'downloader' network context type, which was never fully introduced and wasn't turning out to offer much in the new bandwidth tracking system

version 302

improved how simple downloader parsing formulae are stored and passed around

the edit simple downloader parsing formulae panel now has an 'add defaults' menu button

the simple downloader formula edit panel handles its buttons better and can now do import from/export to clipboard/png and reimport from the defaults, and fixes duplicate names automatically

simple downloaders now remember their current parser through a session save/load

simple downloaders will set a new default parser selection for new simple downloader pages on any choice event

renamed all the default simple downloader parsers to more clearly explain what they do

added 'all files linked by images in page' to the simple downloader, which does the old page of images' behaviour, and is hence pretty decent for most imageboards

added a simple mewch thread html parse formula to the simple downloader

added a very simple webmshare parse formula to the simple downloader

added 'imgur image album' to the simple downloader, which will fetch the first 'page' of results from an image album. full parsing will have to wait for the gallery downloader update

subscriptions can now run simultaneously! you can set the max number at options->downloading page--the default remains 1, so default behaviour will not change

if subscriptions guess they can eat some bandwidth in the next 30s but it turns out they can't, they will bandwidth-override that last job after the 30s expires. this will stop some edge cases from causing subs to potentially hang for hours just on one last request

'explicit tags' are now renamed to 'additional tags'

you can now set media to initially scale to 100% even when it is larger than the media or preview canvas

the html parsing formula can now 'ascend' as well as search down the DOM tree--so, for instance, you can find all 'img' below an 'a' and then jump back up to the 'a' to fetch the 'href' of that image link!

html parsing formulae can now search without a tag name--so, for instance, 'find all tags with class = "whatever"' is doable, without having to specify div or span etc..

the html parsing formula rule panel is hence a bit richer. it also presents a preview of the rule's string as you edit

the client no longer needs lxml to boot, and if html5lib is present, it will prefer that for html parsing. if both libraries are missing and html parsing is requested, it will raise an exception explaining the error

the builds now include html5lib, which is better at recovering from some bad markup than lxml, and as a result some parsing jobs that formerly did not work (such as 'akaiha_(akaihasugk)' on danbooru) now do

rewrote how services test their functional status to better account for different states. the logic here should be a bit better now--repositories will still be able to process if their network side is down, but everything will pause if any pause is set, and it should all propagate up to higher levels of error catching and status reporting better

hydrus can now deal with really big decompression bombs (>~180 megapixel)

filtered out en-dashes (as an alternative to standard hyphens in a "? + -" link-prefix) from booru tag parsing

fixed a string generation issue that was stopping date decode string transformations from summarising themselves

fixed some catastrophic boot failure gui reporting

cleaned up a double-exit error on dialogs that could occur when spamming exit events with a script like AutoHotkey

improved some dead cpu thread clearout code

updated misc old code

misc cleanup

version 301

after discussions with Sankaku Complex about their recent bandwidth problems, added a new 64MB/day default bandwidth rule for sankakucomplex.com--please check the release post for more information

the 'page of images downloader' is now called the 'simple downloader' that uses the new parsing system (particularly, a single formula to parse urls)

the simple downloader supports multiple named parsers--currently defaulting to: html 4chan and 8chan threads, all images, gfycat mp4, gfycat webm, imgur image, imgur video, and twitter images (which fetches the :orig and also works on galleries!)

there is some basic editing of these parsing formulae, but it isn't pretty or easy to import/export yet

the new parsing test panel now has a 'link' button that lets you fetch test data straight from a URL

added a 'gather to this page of pages->dead thread watchers' menu to the page of pages right-click menu--it searches for all 404/DEAD thread watchers in the current page structure and puts them in the clicked page of pages!

cleaned up some page tab right-click menu layout and order

fixed tag parents, which I previously broke while optimising their load time fugg

the new favourites list now presents parents in 'write' tag contexts, like manage tags--see if you like it (maybe this is better if hidden?)

sped up known_url searches for most situations

fixed an unusual error when drag-and-dropping a focused collection thumbnail to a new page

fixed a problem that was marking collected thumbnails' media as not eligible for the archive/delete filter

wrote a 'subscription report mode' that will say some things about subscriptions and their internal test states as they try (and potentially fail) to run

if a subscription query fails to find any files on its first sync, it will give a better text popup notification

if a subscription query finds files in its initial sync but does not have bandwidth to download them, a FYI text popup notification will explain what happened and how to review estimated wait time

delete key now deletes from file import status lists

default downloader tag import options will now inherit the fetch_tags_even_if_url_known_and_file_already_in_db value more reliably from 'parent' default options objects (like 'general boorus'->'specific booru')

the db maintenance routine 'clear file orphans' will now move files to a chosen location as it finds them (previously, it waited until the end of the search to do the move). if the user chooses to delete, this will still be put off until the end of the search (so a mid-search cancel event in this case remains harmless)

the migrate database panel should now launch ok even if a location does not exist (it will also notify you about this)

brushed up some help (and updated a screenshot) about tag import options

fixed a problem that stopped some old manage parsing scripts ui (to content links) from opening correctly

improved some parsing test code so it can't hang the client on certain network problems

misc ui code updates

misc refactoring

version 300

wrote system:known url to find files that have--or do not have--certain types of urls. it works but is still a little slow--I can optimise it later!

added exact match, domain, regex, and url class search types for system:known url

added a button to the top media viewer hover window that will start a file export drag and drop event if dragged from

moved the autocomplete dropdown results list down into a paged notebook

wrote a new 'favourites' page tab for the autocomplete dropdown results

hitting left or right arrow keys on an empty text input will move between the results tabs

hitting arrow up/down/page up/down/home/page or passing mouse scroll events will now go to the current selected page

typing regular search text into the input will automatically return the current page to the search results list

moved the 'tag suggestions' part of the 'tags' options page to a new page

added 'tag favourites' to the 'tags' options page to edit which tags show in this new tab

added import/export buttons to the tag siblings and parents dialogs. they'll export to clipboard or .txt file, and import from the same with an additional option to add_only (i.e. to not delete/petition conflicts with the existing list)

added some quick-and-dirty 'set as alternates/same/notdupes' buttons to the duplicate filter, which will quickly apply that status to the dupes and show some more dupes

sped up db loading time of tag siblings and parents significantly

added a short delay check to tag siblings/parents regeneration so rapid regenerations (such as when processing certain admin-side petitions) can be merged

fixed an issue where similar_to searches could return results not in the current file domain

fixed some spinctrls that were sizing to thin

fixed a bug in the manage server services dialog that was incorrectly dealing with port conflicts on edit service dialog ok

added a clientside and serverside assertion to test that all the services on a serverside modify services call have unique ports

fixed an issue where hydrus network services without access keys would sometimes try to sync their accounts (this was messing up some admin server setup)

fixed some misc dialog window structure

messed around a little with how the autocomplete dropdown hides and shows when in float mode--I _think_ it will now be less flickery and will otherwise position itself and receieve focus better

converted the 'export files' dialog to the new sizing system and also made it non-modal (i.e. you can now interact with the rest of the program while it is open)

wrote a more rigorous force-fit-all-tlws command to the debug menu

misc fixes

misc refactoring

version 299

wrote ui to review and even edit session cookies by network context. it is still a bit rough but will help with future development.

added a 'open_selection_in_new_page' shortcut to the 'media' shortcut set that will work on the thumbnail view

added a 'export_files' shortcut to the 'media' shortcut set that will work on the thumbnail view

fudged manage siblings logic to not do the borked 'hey, that sibling already exists' as soon as you type the old sibling--it will now auto-petition any existing siblings when you click 'add' with a good automated petition reason that makes sense to the janitor

manage siblings now also only shows rows appropriate to the current selection like parents does. it gets a new 'notes' column to specify conflicts that will be auto-petitioned as above

manage siblings and parents now have a laggy 'show all pairs' checkbox to let you quickly review everything like you used to

fixed a database-level bug that meant petitioned and pending the same left-hand sibling tag (like petition a->b, pend a->c) would sometimes not both save together

when you middle-click or right-click->open a new page on a selection of tags/search predicates, the new page will now be named after the tags

if subscriptions hit their periodic file limit, they will now give a little popup message describing what happened and possible causes and actions the user can take

added a 'make a modal popup in five seconds' action to help->debug

modal popups will now hide/show other child frames (like review services) rather than minimise/restore, as this latter action can raise the entire progam to the front

added a 'make a parentless text control dialog' debug entry to test some key event catching

added a 'layout all tlws' debug entry to help->debug that'll hopefully help figure out some child window sizing/position issues

added a 'shortcut report mode', which will report caught shortcut keys and their matched commands, if any

fixed a little shortcut catching bug in the main gui

finished adding the new bytes control

updated ffmpeg for windows

improved sankaku default bandwidth rules to stop a subscription bandwidth rules mismatch that could sometimes make for subscription delays--existing users may like to add a 2GB/day rule for sankakucomplex.com

improved some service and account error handling to better propagate the exact problem up the exception handling chain

the clientside 'update A actually had hash B' repository sync error is now dealt with in a less severe way, and the bad update file is saved to disk with a request for it to be forwarded to hydrus dev for further investigation

file parsing is now more resistant to invalid negative values for properties like width, height, and duration

improved some focus code that may have been affecting linux stability

improved a deviant art url-not-found error

improved a wx version boot test

misc fixes

version 298

wrote a new 'bytescontrol' and related noneablebytescontrol that allows for a uniform way to choose a bytes size value with a wider range--and deployed it all over the place

hitting a mapped shortcut for 'manage file notes' should now ok the 'manage file notes' dialog as well as open it

the manage file notes text control will now start with its caret at the end of the document

manage siblings/parents now state how many pairs they have

as an experiment, manage parents now only lists pertinent information in its listctrl. this is hacky but potentially a big improvement to the workflow here, feedback would be appreciated

added a new menu entry, 'network->pause->all new network traffic', which will indefinitely pause any new network jobs. this value will persist through a restart

'pause subscriptions' is moved from 'services' to 'network'

'pause import/export folders' is moved to 'file' and all the import/export folder stuff is sent down to its own submenu

in lieu of proper session inspection gui, added some debug 'reset login' entries to the network menu for pixiv and hf

if a page has files but none are selected, it will now say the total size in the status bar

the edit bandwidth rules control and its subsidiary dialog use some saner and more user-friendly layout and presentation

the previous search distance on the 'review bandwidth usage' frame is now remembered

fixed some bad logic where a 'copy_bmp' event could trigger despite the current media being None or a non-static image

you can no longer open multiple copies of the subscriptions or import/export folder manage dialogs if you hit the menu entry multiple times while the first is waiting for the jobs to finish

like import folders and subscriptions, manage export folders now waits for currently running export folders to quit before opening

export folders run lighter and quit faster on client shutdown

the domain manager will give a better error if a URL submitted to it lacks a schema (the http or https part)

fixed a bunch of unicode error handling

fixed an issue with similar files metadata orphans prohibiting new file imports

finished up a simple shared shortcut processing object and replaced most old temp duplicate shortcut code with it

misc ui cleanup

misc text rendering fixes

misc shortcut code refactoring

version 297

finished a prototype 'file notes' system. thumbnails and media viewer canvas now support 'manage->file notes' in their right-click menus. this launches a simple text box which will save its contents to db

added 'manage_file_notes' shortcut to the 'media' shortcut set

tag summary generators now have a simple show/hide checkbox and (for thumbnails) custom colours for background and text including alpha channel!

fixed a variety of timing and display logic related to subscription query DEAD vs next check time calculation

all currently dead subscription queries will be revived on update, just in case they were formerly set dead by accident

the 'fetch tags even if url known and file already in db' option is moved from the download/subscription panel's cog icon to tag import options

cleaned up tag import options layout, controls, internal workflow, and help button

added 'select all/none' buttons to tag import options panels with multiple namespaces

if a subscription is blocked by bandwidth, the manage subscriptions dialog will display that in its 'recent error/delay' column

the edit subscription dialog will show similar bandwidth blocking info, on a per-query basis, under a new 'recent delays' column

the review bandwidth usage panel will no longer show some unusual results by default that you can see with 'show all' hit anyway

the review bandwidth usage panel will show the usage at the current search distance in a new column

the review bandiwdth usage panel will show number of requests after data usage. this might be info-overload, so I might alter the syntax or roll it back entirely

fixed an issue with hentai foundry parser pulling images placed in the image description area instead of main image. this particularly affected the artist 'teku'

tags for deviant art and tumblr and thread watchers, which were formerly stored in volatile session memory--meaning half-completed import queues were losing their tags through a program restart--are now saved to the new import object directly

removed all the old volatile session memory patch code

added the new import object through a larger part of the parsing pipeline

deleted the old remains of the giphy parser--if it comes back, it'll all be rewritten in the new system

harmonised some other import pipeline code to the new system

added a new 'management and preview panels' submenu to the 'pages' menu

added an option to control 'save sash positions on close' to this menu

added an entry to force-save the current sash positions to this menu

added an entry to 'restore' the currently saved sash positions to all pages to this menu (this is useful if your window resizes real small and all your pages get crushed up)

rejiggered how URL Classes are matched with URLs to make sure some Post URLs are not lost (this was affecting Hentai Foundry Post URLs, which were sometimes not displaying in the media viewer despite matching)

fixed an issue where the duplicate filter page's jobs would not trigger an update after a job finished

fixed an outside chance of a crash after running a duplicate filter page job

improved how strings are coerced to unicode--now the preferred system encoding will be tried before utf-16, which should improve support for �-type characters in various non-unicode sources (like neighbouring .txt files)

fixed an issue with the client's local booru and flash files (and some other file fetching and mime reporting is a bit faster and neater overall)

the options should be more reliable about redrawing all thumbnail banner summaries on an option ok now

the options->media->media zooms option will now remove any <=0.0 values when it saves

fixed up some old test code

improved how some thread-to-gui update reporting code works

deleted some old network object code

converted manage subscriptions panel to an edit panel--a decoupling refactor I will likely ultimately make across the program

wrote a help page for content parsers

did the first half of a help page for page parsers

misc refactoring

misc cleanup

version 296

the 'allow decompression bombs' option is now moved to 'file import options'. it defaults to False

file import options now allow max size and max resolution rules. they default to None

file import options now allows a max gif size rule to deal with THE SFM COMMUNITY. it defaults to 32MB

file imports will give better quality errors if they fail due to file import option exclusion rules

file import fail errors will now also specify the incorrect amount--e.g. if a file is 64KB but the minimum limit is 256KB, it will state this in the error

wrote a bunch of unit tests to make sure the new complicated file import options object works good

finished the tag summary edit ui and added a new 'tag summaries' page to the options

you can set the thumbnail top, thumbnail bottom right, and media viewer top summaries here, and everything should link up right!

the old options to hide thumbnail top and thumbnail bottom right text are removed from the options->gui page

tag summaries now include pending tags again

tag summaries will order tags according to hydrus numeric tag sorting rules (0 < 0a < 0b < 1 < 9 < 10)

tag summaries will collapse namespace groups that all start with decimals into a "min-max" phrase instead of the whole list (so 'collecting' a chapter will list p1-23 for the page phrase instead of each number in turn)

fixed some potential error popups from the autocomplete dropdown when not in float mode (this hit non-windows users more often)

split the parser help pages up a bit more and updated the client ui html links

finished off the formulae help pages

updated to wx 4.0.1 on all platforms. it promises increased stability!

default tag import options are now moved to the 'importing' options panel, alongside the default file import options

the database will deal with and recover from rollback exceptions more reliably. rollback-loops should be fixed, but the original event will now also produce a request to restart the client (which will safely reset everything) anyway

improved how I create and layout the 'help for this panel -->' help buttons

the job scheduler now regularly checks for and clears out dead wx-related jobs (this saves a bit of memory)

cleaned up an old tag retrieval method that was wasting CPU. some larger pages with many thousands of tags should do tag calculation after selection and so on a bit faster

fixed a thread shutdown test that wasn't working until app shutdown time

improved some more thread shutdown code

deleted some old thread-inefficient calllater and wx-safe timer code

emergency server errors that occur during the response writing stage should be handled better by the client

emergency server errors are more robust in general serverside

improved the quality of the errors thrown by the new network engine's login system

improved some test timing and key press simulation code

misc refactoring

misc improvements

version 295

fixed the runtimeerror popups that would come up on restore from minimise or main gui move after the complete destruction of a general search page

cleaned up some main gui move code generally, and removed a memory leak on the way

file queries can now cancel at multiple checkpoints during the first phase, saving a bunch of CPU time on certain large queries that are replaced mid-search

after a file query has been going three seconds, a little 'stop' button will appear beside the regular autocomplete input. clicking this will cancel the current query! it will stop when it next hits one of the checkpoints above

the floating autocomplete dropdown should be less flickery in some circumstances

dejanked some more file query code

added a 'clear orphan file records' entry to the database->maintain menu. this looks for and purges orphan file rows as you may have seen a notification about recently. this mostly affects the duplicate filter system

fixed up the delete file code to be a bit more robust--it should lead to fewer orphans in future

all the parsing edit panels have new layout: they no longer have info panels but instead a help button that points to the html help, and the edit and test panels are now beside each other rather than in notebook pages

harmonised a bunch of the parser ui test panel code, refactored how the results are stored

the test panel now presents a better 'preview' of what it contains (the actual text control has like 64KB text limit on some OSes and has unreliable text encoding rules, so using it as the raw container for the example data has lead to problems), and we now read and write the example data with a couple of new copy to/paste from clipboard buttons

wrote another new test panel for subsidiary page parsers that does the separation formula stuff a bit better. the test results now come back for all posts as well, rather than just the first

added a new 'deeply_nested_dialog' frame key to options->gui for the parsing ui to better lay out five or six nested dialogs in a nice 'topleft' way

the 'topleft' frame padding is reduced from 50 to 24 pixels to better fit in deeply nested dialogs

misc parsing ui improvements and little fixes

the manage url classes and manager parsers dialogs now have a better 'add defaults' button that allows you to just select the defaults you want (by name) from a checklistbox

wrote a parser for 420chan and added it to the defaults. it should automatically add and link up when you update

if the install_dir/db directory is not writable-to (e.g. you have installed the program to a protected location like "C:\Program Files"), the client and server will default to ~/Hydrus as the db directory

wrote a new 'TagSummaryGenerator' class that will do 'bunch of tags'->'nice summary string' conversions for the thumbnail banners and export filenames

substituted some static tagsummarygenerators to do thumbnail banners

did the same for the media viewer top-center namespace summary

started some edit ui for tagsummarygenerators--I'll have some proper customisable stuff in the near future

moved the background memory maintenance and misc daemons to the new job scheduler, reducing thread count and idle CPU some more

added a debug 'show scheduled jobs' entry for the new job scheduler system

decompression bomb failures will no longer count towards a subscription's fail count, so having a bunch of them won't abandon a sync

fixed and otherwise improved a potential crash condition when a thumbnail panel closes while a menu is popup'd on it

to forestall this program instability, the thumbnail window will no longer replace while its menu is open. the behaviour after this delayed window delivery is slightly borked, but it isn't a crash so I'm ok with it for now

removed some other jank from the thumbnail media panel swap code

non-cancellable modal popups will no longer have the 'close' button. trying to close them with the dialog's X button will still give the 'sorry lad, can't cancel' error

rating and file service system predicates for services that no longer exist will now render a neat 'unknown x system predicate' presentation string rather than throwing an error

searches in 'all known tags'/'specific tag domain' no longer provide system:untagged, wew

some delayed events are now posted in a more thread-safe way

misc refactoring

version 294

fixed video scan

fixed up some import folder logic--they now run 'look for new files' checks separate from 'import anything still in the queue', so they can now catch up on outstanding files more easily

the ten minute file-processing break is reverted--import folders now just 'save' every ten minutes, to forestall lost work on a crash

import folders now have an explicit 'check regularly' checkbox to control whether you want it to check regularly or only when you tell it to. the paused status now means 'do not ever work'

import folders now make a 'working' popup, like subscriptions do, that shows new file discovery progress and found file import progress. it has a cancel button that will stop the current job and can be hidden in the import folder's options. existing import folders will default to ON for this, so you'll likely see one of these right after you update

import folders can now publish their files to a new page! this effectively cuts out the button step. furthermore, if a page already exists with the import folder's name (e.g. a previous page the import folder created is still around), the import folder will publish the new files to _that_ page, updating it

file popup buttons with the same name will now 'merge'! multiple work cycles of a subscription or import folder will now just update the first button rather than spamming several

running import folders will respond faster to a client shutdown event

wrote some better controller-level thread management, including surplus thread deletion--idle CPU should be reduced on import-busy clients

wrote some code to deal with sub-second times a bit better in certain places

finished off a new scheduled job queue that collapses an old multiple-thread-when-idle system down into just one

many things are moved to the new job scheduling system--all the old calllater calls, and the popup message timers as well

completely removed the old wx timers the autocomplete input was using, as they were just too much of a hassle. any crashing these were causing is now gone--it all works on the simpler scheduled job queue now

the autocomplete dropdowns are better at judging when to engage their timers and so now use far less idle CPU

fixed a crash that could occur sometime after starting a duplicate filter maintenance task from the dupe filter page

fixed several unlikely but plausible crashes on the admin-side of petition processing

wrote a bunch of help for the url classes and new parsing system--it isn't finished yet, but then neither is the new system!

if the client fails to initialise the db, it will now try to present the error in a bit of screenshottable-gui before the program quits

improved thread watcher error handling when the given url is unwatchable

lots of timer related cleanup and tiny fixes

misc fixes

version 293

fixed the issue with options dialog not opening again after a save--I apologise for the inconvenience

the default system:time imported predicate (as set in options) is reset to the default '< 7 days'

fixed another potential crash with the manage tags dialog (in fact with any dialog that has an embedded autocomplete or any other autocomplete with a non-floating dropdown list, such as the import file path tagging dialog)

the sankaku user-agent has been updated to be a generic Firefox string "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0", which seems to work with their new rules. if your sank has been broken and you haven't touched the settings that do this stuff, your sank should be magically fixed on update

the default global hydrus user-agent has been updated to the more polite and compatible "Mozilla/5.0 (compatible; Hydrus Client)". if your previous global user-agent is currently 'hydrus client' (the old default), you'll be updated

import folders now deal with interruptions more skillfully and take a break and save their progress every ten minutes

the manage import folders dialog now waits properly for any currently running import folders to quit and save themselves before opening itself (like how the manage subs dialog does it)

import folders deal with some error states in a better way and are able to cancel out of their loops more reliably when an error occurs

added a new checkbox option to options->tags that lets your 'write' autocomplete inputs (ones where you can submit new tags, like in manage tags) select the first result with non-zero count by default. this means it will skip over the first result if it is just nonsense like 'boyp' that you typed to get the results

linked all the final behind the scenes stuff in the thread watcher together with the new parsing system. this stuff remains complicated and non-user-friendly, so please feel free to completely ignore it and the following batch of points:

-

wrote new url classes and parsers for 4chan and 8chan--they should all be automatically linked up on update

the new thread parsers will use the first part of the first post's comment if no subject is set

the domain manager can now cope with URL->API URL links

manage url class links now displays the current expected api pairs of URL class links and filters out what cannot be currently parsed as a result

manage url class links now only displays watchable urls in the bottom listctrl and permits parsers to be linked!

the domain manager can now try to link url classes with parsers based on example urls!

a button to fill in missing links is now available on the manage url class links panel

content parsers that produce urls now define different url types--file, post, and 'next gallery page'

content parsers can now parse a 'source timestamp' type

veto content parsers now use a StringMatch object to do their matching. existing veto content parsers will be updated and _should_ work about the same as before (unless the previous string was crazy in regex rules) using a regex match

html formulas now pick up the 'string' value more reliably in complicated or mangled html

html and json parsing formulas can now work without any rules, in which case they action the top level node (this is useful to replicate the json/html for subsidiary page parsing)

activated some of the new file import objects' features for the new parsing system--now, they store tags and known hashes as informed by what is parsed. this information is tested and used in import

the db can now test all the hash types--md5, sha1, sha256, sha512--during pre-import checking

the new parsing system now takes a temporary 'parsing context' that will eventually be able to receive and serve temporary variables but at the moment just holds a 'url' entry if the parser would like to use the url used to fetch the data anywhere

all the new parsing ui now has a button to edit the parsing context, and the example parsing context is saved to the topmost page parser object

wrote some url classes for 420chan.org--they will be added on update

fixed an issue with the 4chan thread url class

-

the standard new listctrl panel wrapper can now now provide a special 'export to pngs' for certain named objects that will quickly export one object per png

the 'import from png' dialog as launched from the standard new listctrl panel wrapper now allows multiple png selection

the default parsers (and now the url classes as well) are stored in install_dir/static/defaults. they are read from here when you create a db or ask to load from defaults, so feel free to play around with this

the manage services dialog now has an additional Yes/No warning/confirmation dialog if it expects to delete any services on dialog ok

page tabs escape mnemonic characters (typically '&', which in ui labelling rules makes the following character an underlined shortcut) more reliably

in an attempt to stop subs making so many separate small file popups, the subscription daemon will now only naturally check every four hours (previously one hour) and will wait up to ten minutes for bandwidth to free up before dumping out (previously 30 seconds). these values will be user-configurable in a future update

fixed import for some unusual apngs

corrected some more GMT/local time stuff with the new system:time imported 'date' mode

misc GMT/local time fixes (profile logs names themselves correctly, etc..)

improved how the thread watcher reports thread-check errors in unusual situations

fixed an issue generating network contexts when the initial domain is 'localhost'

improved some misc program stability

started a new job scheduler object to reduce idle thread count and CPU usage--will play around with it a bit more and see about integrating it in the coming weeks

version 292

extended system:age to support searching by fixed calendar date, e.g. "system:age > 2018/01/24"

'system:age' is also renamed to 'system:time imported' and presents nicer strings

I believe I have fixed the manage tags dialog crash

the manage tags dialog now loads related tags, file lookup script info, and recent tags off the main ui thread, meaning the dialog itself should launch much faster in almost all situations

cleaned up how a bunch of autocomplete<->manage tags events occur

'set_search_focus' works in the manage tags dialog again, drawing from the new shortcut system

fixed ctrl+scrollwheel on autocomplete entry, which now correctly scrolls the selection over the results again

cleaned up a bunch of the taglist scroll and selection code, particularly for tag children/parent selection events

fixed a potential crash bug initiated by en early exit from the export files dialog

fixed a potential crash bug initiated by an early petition processing page delete

fixed a potential crash bug initiated by an edit repository close during access key fetching

the thumbnail view should be a bit less jittery--the scrollbar should fade neater on mouseover, and maybe the thumbs are smoother as well (I turned off a now-superfluous double-buffering mode, please report any visual bugs)

added 'check_all_import_folders' to main_gui shortcut actions

menus off the main menubar that are out of date (this often happens to 'pages' after a session addition/delete) will now be disabled (greyed out) until they are back in sync

the duplicate filter now detects media it cannot display more carefully. if it runs across undisplayable media due to file orphan status, it will skip over it. if there is nothing else to display, it will ask the user to inform hydrus dev

added json support to the parsing engine!

wrote a new json parsing formula object

wrote a new 'compound' parsing formula object that combines multiple formulae together

wrote new panel for editing formulae with some cleaner button workflow

added multiple 'separated' content parsing solutions to page parsers and wrote all new gui to reflect that

expanded the new separated content parsing solutions to be full-blown nested sub-page-parsers, wew

put some more general work into the new parsing engine--the new page parser object now saves and serialises, so it can be imported and exported. I don't expect to change it much more, but don't rely on that! this is all still under construction

added 'file hash' content type to content parsers and improved ui support for when string conversion rules generate bytes data etc...

new parsing objects now render their parsable content correctly in the 'produces' column

misc cleanup of new and existing parsing dialogs

fixed the handling of escape/cancel events on the 'select from a list of strings' dialog

on update, users with a certain kind of orphaned file will be informed of the numbers and a request to inform hydrus dev

fixed exporting serialisable object to png, which was failing due to a bmp data change in the new wx

fixed a couple of places where I was switching GMT and local time around

deleted some old useless tag editing and commandevent code

version 291

expanded the permissible shortcut keys to all reasonable ASCII characters, such as comma and ampersand and the types of brackets

fixed a delayed crash sometimes caused by the file import frame (the one launched by dropping some files on the client or going file->import files)

the file import frame does its file parsing job in a generally more sensible way, particularly when new drops occur while parsing is ongoing

fixed a crash that could be initiated by the manage tags dialog setting the favourite tags late

fixed a crash that could be initiated by manage parents closing before results came in

fixed a rare possible crash that could be initiated by a review service panel delivering results late

when media pages load, they will be more thread-crash-polite to wx

wrote a new 'PageParser' object that will be the primary parser in the new downloader engine. this can deal with a variety of complicated parsing situations. it is still a work in progress, not yet ready for proper use

started new 'manage parsers' dialog for the new parsing system and arranged a bunch of things around that to make the eventual transition easier

edited some more old 'parsing scripts' ui to again make the transition easier. file lookup scripts will be converted to something new as we move to the new parsing system

fixed some bugs in contentparser instantiation

fixed a bug in the html tag rule edit dialog that wouldn't allow attribute pair editing

the names in the manage tags file lookup script dropdown are now sorted

removed some old redundant error-hiding code from the pubsub system

a common 'is my page closed' check is now a lot simpler and faster

fixed an edge shutdown bug where log messages were not able to be written to the log file

some media viewer objects delete themselves more safely

maybe fixed a rare menu refresh bug in linux

improved how hydrus deletes temp paths--if they are 'in use by another process', it will now put them aside and try again later. this error typically occurs when file mimetype parsing fails, and due to the error context hanging around, an open file handle is still in scope.

the client is better about recovering from missing tags in the db--in certain circumstances, it will now generate a 'unknown tag:random hex' as a simple placeholder that can be better used to investigate the error, rather than dumping out

downgraded os x opencv to 3.1.0 to hoyefully increase compatibility

the server closes a little more neatly

upnp errors are more verbose

misc code cleanup

version 290

content (tag and rating) shortcuts in the 'media' shortcut set now work on the thumbnail view!

tags in lexicophically sorted taglists will now sort numerically if their subtags start with decimals (e.g. page:3 will come before page:20 now)

trying to add/edit a subscription query that is a dupe will now give you a messagebox saying no, and pasting multiple queries will filter out the dupes and inform you (so, if you aren't sure if some of your list of ten artists to add is already in your big 'artist sub', just paste them anyway--it'll take out the dupes for you)

string converters now support regex substitution! (I expect to eventually integrate this into the filename tag parsing panel!)

'modal' popup messages (that create a new dialog and stop interaction with the program, like the one launched for repository processing) will now minimise non-main-gui frames while they work. these frames will be restored when the job is done (this should relieve the problem of this modal dialog sometimes letting other semi-stay_on_top frames obscure it while still prohibiting interaction, wew

simplified and sped up how importers decide whether to do file work or wait

made file importers do ui-polite pauses during the file import loop more intelligently--if the files make no big change to db or ui (e.g file was known already deleted because of its url), no wait will occur at all, meaning redundant lean queue-sections should work real quick

fixed an issue with multiply-nested submenus sometimes not deleting themselves neatly and resulting in highlight-related RuntimeErrors.

fixed an issue where the 'select from list' dialog could sometimes be ok'd without a valid entry--I believe this may be have been causing a delayed crash as well

added a 'ui timer profile' mode, which profiles some common unified ui update loops that draw animations and keep downloader ui synced and so on

the 'delete processed' file import status right-click menu entry will no longer show if it has the same count as the 'successful' line

the options->colours panel has some better layout and the namespace listbox asks a yes/no  to delete if you double-click/enter a colour

dropping a raw file url onto the client will now automatically pend it to a raw url importer (previously, it would be put in the input box, but after rethinking how this could go wrong, I concluded it wasn't such a big deal). if you use this a lot, see how you like it, and let me know if you would want an option to revert to the old behaviour

updated domain manager so it tracks url class identity in a neater way (renaming them will no longer break links or display options)

improved some widget feedback of edit url class panel

simplified the manage url class links panel

url classes now normalise gallery and page urls differently (page urls strip all non-declared data, gallery urls just switch scheme). this may change in future

url classes can now convert a normalised url to an api url--for instance for informing the program that an imageboard thread url has an api version that'll be better to check

the domain manager can now determine arbitrary url parsability capability, and this is just starting to be integrated into url drag-and-drop--try dropping a known url class on the client, like a booru link: it'll now tell you it sees what it is but can't yet parse it!

the 4chan and 8chan thread url classes will reset to defaults again this week--replacing with url classes that convert to the api versions--and you'll get the new api url classes themselves

if missing, the primary tables in client.caches.db can now regenerate on boot. they aren't properly repopulated yet, but the client will boot and you'll get an error message telling you what to do next

improved .txt and clipboard tag import unicode support

removed leading BOM (byte order mark) from .txt and clipboard tag imports

improved some clipboard text fetching code

added a new 'hover window profile mode' to the debug menu--it will print the logic info behind any media viewer's hover window show/hide decision

tweaked the locale-setting code, which was stopping some users (on Win10 creator's update, it seems) from booting the client

added system:rating to 'all known files' query contexts (so you can now search for deleted files that have ratings)

the manage import folders dialog now has a redtext warning about subdirectories

some file importers deal with shutdown events more gracefully

lz4 is no longer required. if it is absent, or the version is too old, a simple statement will be printed to the log and it will not be used in either the client or server

updated to ffmpeg 3.4.1 on windows

updated opencv to 3.4 on windows and linux (had trouble with os x--will continue working on it next week)

fixed an important grammar typo in the help

version 289

fixed an issue where scrollbars were only appearing on taglists after a resize event

fixed the raw filename component of file drag and drop events from the client to external programs

fixed the tag lookup scripts

fixed some wx menuhighlight issues

improved some shutdown code

fixed the add/edit namespace colours options panel, which needed to be updated to deal with the new wx's better alpha channel reporting

fixed an issue when hitting 'open externally' on a media collection

fixed a crash on client shutdown whenever closed pages were in the undo menu

think I fixed another shutdown crash

fixed a rare issue with the collect by dropdown not being able to generate a string to display

misc wx fixes

added a 'importing' page to the options dialog, which now sets the default file import options for quiet and loud file import contexts

the old and no longer used 'exclude deleted files' option is now removed from the 'files and trash' panel

finished off default url matches for all downloaders that come with the program--these will be set on update, so if you have custom ones, please export them before you update so you can import them again!

improved how urls are matched and presented for the user in the media viewer

added a 'delete "successful" file imports from the queue' entry to the file import status button right-click menu--this only removes 'successful' and 'already in db', leaving anomalies

improved locale instantiation in the client and added locale strings to the help->about dialog

you can now set the page name prefix for a paused thread checker. it defaults to a unicode pause character: ?

thread watchers will no longer pause on a network error during a check--they now have a 'delay' system like subscriptions, and on a network error, they will delay four hours (unless you hit 'check now')

patched in some simple 'connection cutoff' network error handling, we'll see how it does

wrote in some more proper error handling for a specific connection cutoff error that is being produced

the similar files search tree regen code now clears out orphaned files. if you have had blank 'unknown' files appear in similar files searches, please run database->regen->similar files search tree

bitmap buttons on download pages will now update using less CPU and will flicker less

improved some video rendering error reporting

fixed the 'author name' regex favourite default, which had a superfluous asterisk. if you would like to fix it yourself, please try: [^\\]+(?=\s-)

added 'flush log' debug command

client.pyw now makes a safe 'system' ui error popup if it fails to boot

version 288

updated to wxPython Phoenix (4.0) build!

all builds now require the new wx, so if you run from source, please consult the help files for new info on figuring this out

did a ton of wx refactoring

merged a large number of timers

wrote a new wx-aware timer to replace some of the more awkward old timers that could not be neatly merged

replaced wx.calllaters (the new ones aren't always garbage collecting nicely) with the new custom timer object

misc wx deprecation refactoring

cleaned up some wx test code

fixed some crashing wx test code

fixed 'select from list of strings' object, which was no longer processing enter key properly

cleaned up a bunch of ui object interaction code

cleaned up a bunch of wx-related shutdown code

fixed an issue with the splash window not shutting down cleanly

fixed a shutdown db-gui status report crash

was unable to get flash embed windows working properly without edge-case crashing on windows, so they are disabled for now (you'll likely get an 'open externally' button instead)

animations now clean up their memory buffer faster in certain circumstances--all users who view animations in the preview window should notice significantly leaner 'idle' memory usage over time

fixed a very important inefficiency bug that meant select->inbox/archive on large pages with more than a handful of inbox files was taking extremely long periods of 100% CPU

wrote a faster way of fetching some media and their paths

copying files to clipboard is now much faster

initialising any large file drag-and-drop event is now much faster

certain redundant image pre-fetching will no longer be done

'open externally' panels will no longer draw a (slightly buggy and useless) animation bar if the underlying media would have had one

added a new 'menu profile mode' that will profile any menu click

cleaned up the debug menu

deleted some old options code that is no longer used

version 287

thumbnails can now be drag-and-dropped to other pages!

dragging and dropping thumbs to another page tab will remove them from the source and append them to the destination, maintaining file order!

DnDing thumbs to a 'page of pages' tab will put the files in the next lowest selected media page

DnDing thumbs to a blank notebook area (or a page of pages without a selected media page) will create a new page for the thumbs

holding down ctrl when you drop thumbnails will not remove them from the source

please forgive the thumbnail DnD cursor, which for now will be in the 'copy' state, despite the internal DnD being move by default

improved page tab drag and drop drop logic--dropping onto the page area itself will no longer send the page to the right-end of the current notebook

the 'file import options' object now supports three 'presentation' booleans--for new/already_in_inbox/already_in_archive files--so you can customise whether new thumbnails appear based on each state. page imports will by default show everything, while 'quieter' import queues like import folders and subscriptions will continue to just show only 'new' files in their files popup buttons. if you have a gui page with 10k+ items in its queue, try reducing the presentation to speed it up!

all existing import queues will be updated when they are next loaded--but please note that for simplicity and safety they will all initialise to the 'quiet' presentation method, so if you have ongoing download pages in any of your gui sessions (including thread watchers!), they will only add 'new' thumbnails unless you edit them. I apologise for the inconvenience

the regular hdd import now has a file import options button!

subscription query 'finished' file popups are now merged up to the subscription level--so, a sub with five queries that each get 20 new files in a run will now ultimately leave one popup with 100 files

file popups (as produced by subscriptions and a couple other places) now preserve their import order!

if a subscription with many queries runs out of bandwidth, it should now only give you one 'no more bandwidth to download files' message, rather than one for every outstanding query to sync

added a checkbox to turn on/off the new random subscription syncing to the options->downloading panel

the file import status button's menu now supports import/export of sources to clipboard/png! it _should_ also support unicode. be careful not to paste paths into a url cache, or urls from one gallery site to another, or you'll just get errors--this is prototype, so please keep like with like for now

the png import/export system now supports raw string payloads

the new listctrlpanel can now hang its buttons in multiple rows

the manage subscriptions panel now has an 'overwrite checker options' button to mass-set checker options

the manage subscriptions panel now has a 'select subs' button that selects subs based on a basic query text search

separating merged subscriptions now sets better new subscription names of 'old_sub_name: query_text'

saving a session from a page of pages with a custom name will no longer suggest a session name prepended by [USER]

doubled the subscription and downloader instance default bandwidth rules to 400 and 200 rqs/day

the 'load_images_with_pil' and 'disable_cv_for_gifs' options are now officially BUGFIX in the options--unless you know you need them on, turn them off!

added some safeguards to the new dialog-panel system's OK stuff, which sometimes catches a duplicate OK event

shuffled some db update status texts around

version 286

simplified how thread watcher assigns DEAD and 404 status--it should do it more reliably now

thread watchers now publish their page names (including updating dead/404 status) right after they check

the save session menu under pages now lists the exsting sessions for quick-save--if you select one of these, it will throw up a yes/no overwrite confirmation dialog

'appending' sessions now loads them in a new page of pages named as the session name!

right-clicking on a page of pages now gives you a 'save this page of pages to a session' option

right-clicking on the page tabs lets you append a session anywhere

'load' session is now 'clear and load', and it throws a yes/no dialog for confirmation

the gallery pending queries box now allows multiple selection with ctrl/shift clicks. it'll even move up/down correctly on non-contiguous selections!

the gallery query input is now a new combined textctrl-and-paste button control that will appear in other places across the program

the page of images input now also uses this new control

the page of images and raw urls queues will now ignore any input that does not start with 'http'

pasting a list of texts from the clipboard will now typically strip the pasted content of leading or trailing whitespace

some text and tag cleaning is a bit faster and neater

the file import status panel now supports 'open sources' on its right-click menu--it opens in your web browser or file explorer as appropriate. (linux can't do file explorer though)

thumbnails will no longer 'fade' when their pages are not visible, reducing CPU load in several contexts

thumbnail fade will use less CPU overhead on very fast computers

the timer loops that all import pages use to keep themselves updated are now harmonised into one loop at the top gui level (this should reduce some idle CPU overhead and improve some ui responsiveness)

content (tag and rating) shortcuts can now be set to 'flip on/off' or 'set'

the manage subscriptions dialog now waits for any currently syncing subs to cancel and save themselves before opening itself (keeping it all synced and rollback-proof)

added a 'watchable' url class type and wrote examples for 4chan/8chan thread urls

added 4chan/8chan file url classes to the examples, which if added will auto-hide them from the media viewer

fixed 'add parameter' in edit url match panel

simplified some stuff in the existing parsing system

added preliminary 'url type' support to the parsing system

the html formula parsing object can now return the full child html of the node it finds

the new url and html stuff doesn't do anything yet--but it will in the new downloader engine

added a bunch more booru entries to the url classes defaults. I will continue filling these out, hopefully setting comprehensive defaults next week

the advanced mode thumbnail right-click menu lets you reparse and regen thumbnails for any files! if you have rotated or stretched files, you can now fix them from the menu!

added new tools to let the client update media file metadata and thumbnails live!

the new reparse and regen should update file metadata and thumbnails live!

improved how some 'quiet' errors are printed to the log. they'll now have the full trace of the error as well as the stack

added a daemon_report_mode debug mode. it throws up a popup every time a daemon fires its callable

fixed an issue with the editstringtostringdictcontrol

fixed an issue with loading (and updating to a new version) import pages or import folders with some kinds of unicode paths

you can now set the 'woah, you should close some pages' warning value under options->gui. its default remains 165 pages

system:similar_to searches should be a lot faster on clients with many files

the manage subscriptions and edit subscription panels now both list their num_urls summary column as '22' if all done but '11/22' if not. this column also sorts based on percentage completion, then num total, then num done

the database migration dialog now uses the new listctrl--its buttons are now also clever, and will disable when they are invalid

misc fixes

fixed an index bug in the new listctrl after certain types of 'setdata' call

cleaned up some search code

some veto code cleanup

cleaned and harmonised how text is pulled from the clipboard

version 285

added 'network' main gui menu and moved network stuff from services to it

split the new domain manager stuff up into separate dialogs and menu entries on the new network menu

manage url classes dialog now lists url type in the listctrl

added url class links info (which will permit client-specific settings and downloader mappings for url classes) to the domain manager

wrote a 'url class links' dialog and added it to the new network menu (only the 'display on media viewer' part works atm)

the domain manager now filters urls on the media viewer depending on whether they have a url match and are set to display in the new links panel

updated the local booru service code to the new service system

the local booru's shares can be reviewed again under review services

the local booru's port and bandwidth settings can be set again under manage services

the different gui parts of the local booru are updated to new controls

fixed a local booru 404 reporting error

the edit subscription panel now has a 'paste queries' button that lets you add queries en masse

added 'manage_file_urls' to shortcuts system

added several 'get_similar_to_x' actions to the shortcuts system

the manage upnp dialog now initialises its mappings on another thread and fails better when UpnP mappings cannot be fetched

connection and readtimeout network exceptions are now recognised more specifically by the client. subscriptions will only delay for an hour on one of these exceptions

improved the resilience of the HF login code after wake from sleep (when networking is often not available for a couple of seconds)

like the recent subscription query randomisation update, subscriptions themselves are now synced in random order (this stops a subscription named 'aadvark' always getting first bite into available bandwidth)

fixed import for jpegs that have unparsable exif data

fixed a bug in 'next month' bandwidth estimate calculation when the month is December, wew

fixed some logic that was setting max page at 165 rather than the intended 200 and added a dialog that even lets you engage the debug override at 200 if you are insane

all audio mime detection and duration parsing is now done through ffmpeg. hsaudiotag is no longer needed to run the program

since the old listctrl sort crash bug is still hitting some people, I've disabled sort on the old listctrl class. feel free to try to sort any listctrls now, no matter your situation. I will continue replacing the old class with the new working class over time

updated another listctrl

a ton of misc controller options/manager access refactoring

cleared out some old code

moved Time controls to their own file and added velocity and checker options stuff as well

wrote a new edit panel for single controls and updated the time delta button to use it

misc refactoring

version 284

fixed subscription queries turning dead on the initial sync

all dead subscription queries have been set to check again in case they can revive

added query file velocity to edit subscription panel

subscription network contexts now reflect the new multiple subscription query system, and are named "sub_name: query_text". as every query now counts as its own separate subscription network context, this will stop query-heavy subscriptions from throttling so much on bandwidth limits

finished URLMatch object, which matches and normalises URLs into certain 'classes' like 'gelbooru post url'

expanded some URLMatch subdomain options

fixed some test logic in URLMatch

finished the last of the EditURLMatchPanel

split the 'manage network rules' dialog into two panels--it now has a 'url classes' tab

wrote a panel for managing url matches

added export/import/duplicate buttons to EditURLMatchesPanel

wrote some URLMatches for hentai-foundry as an initial test of the system and added a temp button to add them--please check them out to see how it all works

all, invert, inbox, and archive (and none, lol) thumbnail 'select' menu items now have counts

invert is now at the bottom

the thumbnail select menu now has local/remote entries if applicable (this typically is only true in 'all known files' file domain)

added png/clipboard export/import/duplicate code to the generic new listctrl button wrapper panel, which will save a bunch of time as the png/clipboard sharing system expands

added a human-facing serialisable name to all objects on the new serialisation system and tied the new import/export code into it for png presentation

the edit import folder dialog will now complain (but not veto) on an ok event if any of the entered paths do not exist

if you attempt to manually run an import folder while import folders are globally paused, you'll get a little popup telling you so

added an experimental 'thumbnail fill' setting to options->gui. it zooms the existing thumbnails so they fill the whole thumb space. feedback on this from those who would be interested in a prettier system would be appreciated

added 'paste tags' buttons to filename tagging options panel

the paths/urls in the file import cache are now their own object that holds the creation/modified/source times and current status and note. this object can also hold prospective urls, tags, and hashes for future use

a bunch of file import actions are faster

all the different importers now use this new file import object

fixed a screen position calculation in the new drag and drop filtering code that was accidentally including too many possible drop candidates on drops in the top-left corner of the main gui (if you had trouble moving tabs to the left, this should be it fixed!)

fixed a problem display volume/chapter/page tags that included unicode characters in thumbnail banners and media viewers

fixed a rare media display bug in the dupe filter

fixed some 'C++ part of panel has been deleted' bugs in review services if the frame is shut down before delayed db info is fetched

cleaned up some more 'C++ deleted' errors in import files selection dialog

fixed the network context custom header panel 'add' action, which wasn't saving the value of the panel

fixed a bunch of bugs in the newish QueueListBox class

SynchroniseRepositories daemon will be better about quitting early on application shutdown

cleaned up some pending pretty timestamp grammar

when the client cannot clean up a temporary file, it will print more error information

added pylzma to the 'running from source' library recommendations. this is not required, but if available it adds ZWS flash support

version 283

subscription popups show a bit more info about their individual queries

subscription popups hide the sub-query info in text and file buttons if it is the same as the subscription name

subscriptions now work through their queries' downloading phases in random order (so a heavy query named 'aardvark' won't repeatedly choke the sub's other queries if bandwidth is tight)

if a subscription query dies, a notification popup will now be made

changed a logical edge case that may have been killing some subscription queries after the first sync

re-enabled pixiv artist search, as it still uses the old gallery style

added tentative support for 'ZWS' flash files

if there is more than one import folder, the 'check import folder now' submenu now has a 'check all' entry

TimeDelta controls that have the 'days' entry (as in thread/subscription checker options panels) now have a max period of 10 years for those _really_ slow threads

fixed an issue where the 'quick namespaces' section of the new filename tagging options panel was not initialising correctly when it loaded data from a previously serialised import folder (this was the "unhashable type: 'list'" bug in the edit import folder panel)

the filename tagging dialog now works on the new panel system. it also has an entry under options->gui: 'local_import_filename_tagging'

polished StringMatch object

polished EditStringMatchPanel

html formulas now have a string match object

improved parsing error system

miscellaneous improvements to the parsing system

misc fixes to StringMatch and parsing system

the main gui will no longer interpret text drops on child frames as url drops. this fix just quashes the drop--it doesn't allow the initially desired drop on the child frame to occur. I will keep thinking about this

the dupe filter's 'no media to display' text now says 'Looking for more pairs--please wait'

the 'remove' media action is now labelled 'remove from view' to reduce 'is it a delete?' confusion

fixed an icon initialisation order problem that meant pre-boot errors were unable to display

fixed a graceful-shutdown-on-boot-error bug

the custom 'listboxtags' controls now post an event on their contents changing

the tag censorship panel now updates its status text when its listboxes change due to a mouse double-click event

fixed file import 'source time' column sometimes showing '3e-05 years' kind of garbage that was due to a missed float->int conversion

cleaned up how some page names are thrown around, which should stop some stray [USER] prefixes slipping though

the adminside petition management checklistbox will now escape mnemonic characters (typically '&') and hence display them correctly on sibling pairs and so on

wrote a 'help my client will not boot.txt' that explains the new debug builds and some other common fixes to try

'all known files' file search domain is now hidden behind advanced mode

fixed another instance of flashwin causing a failed boot

cleaned up some old broken unit tests

some misc help improvements

updated Windows build to new version of sqlite

version 282

rolled back to an older version of pyinstaller that does not seem to have the embedded flash window problems windows users experienced

added an error handler to wx.lib.flashwin import--if it fails to import, the client will print the error to the log and thereafter show 'open externally' buttons instead of the embedded flash window

created a 'filename tagging options' object to handle the instance-non-specific auto-tagging parts of the filename tagging dialog

moved all the appropriate tag generation code to this new object

extracted the simple and advanced panels in the filename tagging dialog to a separate panel

wrote a new wrapper panel to edit filename tagging options objects directly

cleaned up some tag generation code--it'll now all be siblinged/parented/censored the same way

import folders now use the filename tagging options object, harmonising this tag parsing code

edit import folder gui now has an add/edit/delete listctrl to manage the new tag_service->filename_tagging_object relationship (this replaces the old .txt management button and summary text)

finished the StringConverter class in the new parsing engine. it applies a series of transformations to text

wrote a panel to edit string converters

wrote a panel to edit string converters' individual transformations

updated html formulas to use string converters instead of the old cull_and_add system

html formula edit panels can now edit their new string converters

file lookup scripts now 'stringconvert' their file identifier strings--this should allow the 'md5:md5_hash' fix we've been talking about (i.e. by first encoding to hex and then prepending 'md5:')

the help->about window now shows the client's default temp directory

thumbnail regeneration--particularly full-size regen and resized png gen--should be much faster in the client

the debug exes are now included in the windows build

the debug exes are now included in the non-windows builds

the test exe is no longer included in the windows install (can't remember it ever being useful, and it is like 10MB now)

unfortunately, a recentish change to how pixiv serves gallery page results has broken the hydrus pixiv downloader. this will have to wait for the downloader overhaul, and even then, it might be _slightly_ tricky. pixiv downloader entries are now hidden, and existing pixiv subscriptions will be paused on update

the thumbnail select submenu now clarifies 'all (all in x)' if all are in the inbox or archive

you can now archive/delete filter from the thumbnail menu on a single file

lowered the http session inactivity timeout even more, to 45 minutes

fixed a couple of instances of the new subscription dialogs being unable to mass-retry failures

ffmpeg parsing errors now print more info about broken file path

some daemons will be snappier about shutting down on application shutdown

took out the sometimes invalid 'booting' phrase in the during disk cache population status report

the client will now warn you about page instability at 165 pages after one user reported instability at 175. 200 is still the strict limit.

downloader pages and subscriptions will fail more gracefully if their downloaders cannot be loaded (e.g. they used a since-deleted booru)

fixed listctrl panel button enabled/disabled status when the child listctrl starts empty

the new listctrl can now select data in a cleverer way

misc fixes

misc refactoring

version 281

subscriptions can now support multiple queries!

subscriptions now use the new thread checker system to set their check periods!

improved some check timing calculations to deal with subscription timestamps

existing subscriptions will update

subscriptions can now see any 'delay' reason on their edit page and can scrub it there as well

subscriptions no longer track 'last error'. everything goes through the delay system and keeps track of reason

added 'last new file time' to the subscription listctrls

subscriptions now generate separate 'file button' popups for each query, and file button popups now include their name in the button label

the subscription listctrl is now breddy wide. I will be adding some way of hiding columns (with persistent memory) on the new listctrl at some point in the future, so please bear with it for now

manage subs now has a wrapper panel and buttons that disable when invalid

tumblr simple 'video' posts should now be supported! turns out they are just mp4s

in the client, static image thumbnails are now generated with opencv

files that give truncated image errors should now import ok. they'll render with errors. if cv is up to date and working ok, they should have good thumbnails as well (otherwise they will have all-black thumbs)

updated manage regex favourites dialog to the new panel system

regex favourites are now editable under the new options->regex page

ongoing import or export folders will now stop working if their respective manage dialogs are opened

import folders can now stop working if the client shuts down during the initial file parse phase

export folders will now stop working immediately if the client shuts down

export folders can now stop working during the file copy stage

the splash screen is now a little wider and taller

the splash screen now has a subtext row for misc technical data

the boot and exit splash screens show a bit more data about several things

the repository sync daemons should be a bit snappier about responding to a client shutdown

the import files dialog is now a non-modal frame! (you can now continue to use the client while this frame parses files) you can even open up several at once!

the 'import file path tagging' dialog should now never be taller than its spawning screen and also slide up and left if it spawns with its bottom-right corner off-screen (this dialog is still on the old system, however, so the newer sizing stuff that does this stuff automatically is not yet available)

sliding up-and-left should now also occur on spawn for frames on the new system

the advanced 'process now' repository button will now disable itself if there is no work to do!

if simple downloads like gallery pages are larger than 100MB, the client will throw an error (this is a stopgap for an ongoing issue)

the 'maintenance and processing' page has a bit more warning text

the new listctrl wrapper panel now supports menu buttons

libpng's iccp terminal warnings from loading the flash/pdf/zip thumbnails should now be gone!

misc refactoring

version 280

the client should now recognise EXIF file rotation and flipping in resolution discovery, thumbnail generation, and image rendering. existing files will be borked, so will have to either be manually deleted&reimported, or wait for a maintenance routine that will retroactively fix all these

videos with a non 1:1 sample aspect ratio (wew lad) should now import with correct resolutions

all 'tag import options' are now managed with the new button

deleted all old tag option management gui, including deleting ClientGUICollapsible entirely

with the simpler tag import options, increased max page count to 200

page of images downloader now has two pause separate buttons for queue and files processing

the page of images downloader should no longer too-quickly blank out its parser status at the end of queue processing

the default hydrus bandwidth rules now no longer has the choking 'requests per day' limitation that was messing up large mappings uploads. (it is now just 64MB/day)

your hydrus default bandwidth rules will reset to this on update

POST requests (which in for our purposes are always user-driven) no longer obey bandwidth rules

fixed a bug when sometimes hitting the file limit or hitting a siteside 404 in the gallery downloader

a rare bug due to empty string menu labels is now no longer possible. empty string labels, if they slip through, will be replaced with 'invalid label'

you can no longer choose the empty string as a gui session name

improved how network jobs keep track of all applicable network contexts vs the preferred session and login contexts

cleaned up a bunch of login code, improved verification testing

fleshed out domain login system

in the networking engine, the read timeout is now six times the connect timeout--we'll see if this reduces some hydrus and other read timeout problems we've seen

increased the server transaction period from 10s to 120s--we'll see if this helps reduce some spiky server POST lag or what

the default tag import options panel now edits on double-click, rather than deletes. it also yes/no dialogs on a delete action. this control is still trash, but it now works more like the rest of the program

version 279

moved the hydrus service object over to the new networking engine

hydrus services now appear in the bandwidth review panel

hydrus requests now obey larger network bandwidth rules (this mostly means 'global')

service bandwidth usage and rules are no longer managed from manage services--it is now under review bandwidth usage, like all other network contexts

updated some network engine stuff for misc hydrus states like 'server busy'

fixed a bug in the server code where the session key cookie had an invalid expiration timestamp--servers should update this week to make sure new clients can log in properly

hydrus network requests will force-set User-Agent as 'hydrus client/(network_version)' as the network version is used in the protocol to determine compatibility

fleshed out the hydrus specific network job, giving it the various bandwidth tracking and version checking responsibilities the service object used to have

moved session cookie decay (only matters for Hentai Foundry atm) to the new session manager (was previously hacked into the login manager)

moved some hydrus response parsing stuff around, added content type awareness to the new network job

updated several unusual hydrus 'static/test' requests used to test credentials and fetch access key and so on to the new system

added a special 'test service' service to better accomodate these requests

wrote a static login script for hydrus services

polished up login management system overall

import folders now support a 'check now' state, like subscriptions, that will cause them to check immediately

import folders can be 'check now'ed from the file menu under a new submenu! if you would like to have a 'manual' import folder, try pausing it and just running it from this menu!

added an optimisation to the file search algorithm to search ratings queries super fast when they lack tag or file system preds to otherwise speed them up

updated the booru presets that now support https to be https

added a 'restore defaults' button to the manage boorus dialog--you can restore specifics or all of them

optimised how fading thumbnails are blitted to screen, which may provide a huge performance boost for high-res/small-thumb clients

pages that have been renamed by the user will no longer be rename-overwritten by any auto-renaming system (currently just thread watchers, I think, but this will expand in future). unfortunately, this is not retroactive--only pages renamed from now on will be aware that they were user-renamed

in advanced mode, the pages menu now states how many pages are currently open

the 'page of images' downloader will now say '(x already in queue)' when it reports how many urls were found, if any were already in the queue. (this should clear up some confusion where it would previously say '0 new urls' even when it found some stuff)

the gallery downloader will do essentially the same, but on a per-page basis. the text is a little crushed, so I may revisit this

fixed an issue with the manage services dialog not being able to rename dupe-named services on edit subdialog ok

the manage services dialog now uses the new listctrl! the listed services are no longer a horrific unsorted mess!

fixed the file import status button not showing on raw url downloader or import folder edit dialog

improved how some directory tree merging code deals with read_only files in the source

maybe fixed some unusual selection behaviour with the booru selection popup dialog. it now has a real ok button, rather than a mickey-mouse hidden one

completely deleted the old networking engine!

cleaned out some unused imports from the networking code and related entries from running_from_source.html

changed up and fixed some odd bugs in how how repositories test some error/isfunctional stuff vs regular paused status

hydrus network contexts now have a prettier name

ip address-based network contexts will no longer spam the bandwidth tracker with their useless subdomains

network jobs that are unable to attempt validation or login will now error immediately rather than waiting indefinitely

fixed up a bunch of test code that was broken due to the mismatch of network engines

broke some other test code due to the network engine transfer!

the client is more resilient about broken 'twisted' installs, and _should_ be able to boot without it. this may or may not apply to the built release--more work can be done here

some network job refactoring

clientside service code cleanup

misc fix that I can't remember

misc cleanup

misc refactoring

version 278

fixed the tumblr raw url converter to now point to the data.tumblr.com domain

added a hardcoded ssl verify exception for data.tumblr.com, which has an incorrectly defined ssl cert (at least for public-facing interactions), wew

all existing db urls and file import cache urls for media.tumblr.com will be updated to data.tumblr.com on update! (everything _should_ just magically work again)

fixed apng import, which the decompressionbomb detection code was not handling correctly

collapsed the different instatiations of the 'file import status' button down to one class

the file import status button now has a right-click menu that supports 'retry failures' and 'delete processed', if applicable

misc import status cache cleanup and refactoring

you can now edit or completely turn off the [404] and [DEAD] thread watcher page name prefixes under options->downloading

thread watchers should more reliably keep 404 status

'open selection in a new page' now preserves file order!

'view this file's duplicates' now sorts the files!

options->gui now has an option to change how often 'last session' is saved

'last session' will no longer autosave to the database if there are no changes

tags exported to neighbouring .txt files are now correctly sibling-collapsed

tags imported or exported via neighbouring .txt files are now correctly tag censored

the manage tags dialog will now protest with a yes/no dialog if you attempt to cancel it with uncommitted changes

the manage parents and siblings dialogs will now protest with a yes/no on an ok event if there are 'uncommitted' pairs in the lower boxes (e.g. if you forgot to click the 'add' button)

fixed an issue that would sometimes stop old sessions from loading properly

the duplicates page now does its maintenance jobs in modal popups!

attempting to apply a duplicate status to more than 100 pairs now throws up a warning yes/no dialog

the manage urls dialog now has copy/paste buttons

added a (somewhat debug) option to disable the mouse hide&anchor behaviour on slow Windows canvas drags to options->media

added a 'regen all phashes' command to the database regen menu

the disk cache options in help now have a help button to explain good values for ssd vs hdd

the edit bandwidth rules dialog now uses the new listctrl

merged the old and new login managers

misc login work

misc refactoring

misc cleanup

version 277

expanded the domain manager into a legit serialisable object that holds data and saves itself on changes. to begin with, it supports custom http headers for particular on network contexts

the global User-Agent (which now defaults to 'hydrus client') is set through this new manager, as attached to the 'global' network context

wrote a basic panel to edit custom http headers under services->manage network rules (this panel will be fleshed out with more in future--please ignore if you are not an advanced user)

wrote a panel to edit an individual custom http header

wrote a panel to edit/select network contexts. this will get a bunch more use elsewhere as the overhaul continues

flushed out and tested the new popup yes/no button system

sanakucomplex.com has a User-Agent entry in the new custom header system--on the first sank request, you will be presented with a yes/no popup asking if it is ok to use.

if you ok the user-agent, sankaku now seems to work again!

sankakucomplex.com network context will get some specific conservative bandwidth rules (80 rqs per 7m, 1 rq per 4s) on update

added an option to options->gui to reverse the page tab shift behaviour

'don't scroll down on key navigation event if thumb is at least this % visible' value is now 75%. it is also editable under options->gui

the hydrus icon used on frames is now the non-transparent version that shows up better on dark backgrounds

the standard hydrus.ico used in web favicons and the executable builds is now also non-transparent (this should also propagate up to OS shortcuts and taskbar icons wherever the frame icon is not inherited)

file export drag and drop events will now defocus the currently focused media on successful drop (e.g. if you drag a video to an external media player, it will stop rendering clientside), just like for open externally

database analyze maintenance should be more reliable with respect to fresh repository syncs

reduced default impermanent session timeout in new networking engine to 60m (should fix/reduce some hentai foundry 503s)

misc ui improvements and speedup

misc fixes

version 276

the new thread watcher object will no longer produce check periods shorter than the time since the latest file. this effectively throttles checking on threads that were posting very fast but have since suddenly stopped completely

thread watchers now parse their thread subject and place this in the left management panel

thread watchers now name their pages based on the thread subject, if one exists

an option to permit or deny thread watchers renaming their pages is now under options->downloading

dead and 404 threads now disable their checker pause button--to attempt to revive, hit 'check now'

thread watchers now preface their page name with [DEAD] or [404] when appropriate

misc thread watcher code improvements

added basic import support for zip, rar, and 7z files. they get no useful metadata (yet) and have a default 'archive' thumbnail

the client will now by default detect and not import decompression bombs before they blat your computer. an option to allow them nonetheless is under options->media

the server will now not parse or accept decompression bomb uploads in POST requests

added a 'refresh all pages' entry to page of pages's right-click menu

added 'send this page down to a new page of pages' to page right-click menu

added 'send all pages to the right to a new page of pages' to page right-click menu

fixed a page of pages drag and drop issue when dropping the last page of a notebook onto the same notebook tab

fixed some index calculation problems when DnDing page tabs to the right on the same notebook

sending a refresh event to a 'show selection in a new page' page (which has no search predicates and so cannot 'refresh' its search) will now trigger a sort event (like importers got last week)

thumbnails at the bottom of the current view but are at least 90% in view will no longer scroll into view when selected

click events will no longer scroll thumbnails that are semi-out of view into view

improved how all 'wait until the client ain't so busy' checks work. importers that have a whole slew of 'already in db' to catch up on should now not clog the gui so much

similarly, under ideal conditions where nothing is busy, importers will iterate over their files more quickly

the network engine now has a 'verification' loop that doesn't do anything yet, and a stub domain engine is generated to be consulted in this

wrote some verification code, extended popup messages to support yes/no questions

polished some domain engine code

fixed an issue where file repositories were not recording deleted files in certain cases

all file repositories will be reset on update

the date entries on the review bandwidth bar chart now have leading zeroes on 0-9 months to ensure the sort correctly (this month's 2017-10 entry was sorting before 2017-8, wew!)

the migrate database dialog now shows approximate total thumbnail size

gave the migrate database help a quick pass

gave the 'help my db is broke.txt' file a quick pass

version 275

if you hold shift down while dropping a page tab, the client will not 'chase' that page to show it (try it out!)

the gui will be more snappy about dealing with drag-and-drop drops (of types file import, page tab rearrange, and url drop), returning the mouse to normal state instantly on drop and dealing with the event in a subsequent action

dropping url text on the client will specifically inform the source program not to delete the dragged text (that the operation was copy, not move), if it is interested in hearing that

page drag-and-drops should transition a little less flickerily

all file import status objects can now track 'source time', typically to represent upload time

file imports now populate 'source time' based on the earliest of creation/modified time! (this will be used later to parse as a tag)

thread watchers now populate 'source time' based on post time!

finished a watcher options object for the new thread checker system

wrote a panel to edit watcher options

converted the thread object to the new watcher system

thread watchers now have two pause buttons--one for the file queue, one for the checker

compressed thread watcher ui layout

converted the thread left-panel ui and options->downloading page to reflect the new watcher system

improved the watcher options to generate better timings for fresh threads

cleaned up some thread watcher check time code

the total/selected mime summary on the status bar is a little prettier and will now report by individual mime sometimes

generating the total/select mime summaries are now faster on pages with >1,000 files (it'll just use 'file')

a 'refresh' action on an import page now triggers a sort event

added 'flip_darkmode' shortcut command to 'main_gui' and 'media_viewer' shortcut sets

added copy_file/path/hash/bmp actions to the 'media' shortcut set, removed the hardcoded ctrl+c for copy_file. I added ctrl+c to the defaults, but existing users will have to re-add it manually if they want it!

simplified some page ui update code

import pages will no longer update their left-panel ui (which uses a bit of cpu) when they are not in view

polished some new string and url matching code the domain engine will be using

wrote a panel to edit string match objects

wrote a new panel to handle simple ordered lists of data in a better way

wrote most of a panel to edit url match objects

misc domain manager work

fixed an issue with the old listctrl where object name de-duplication was sometimes not permitting (1)-type names to be cleaned up

fixed some inelegant time duration->text conversions

improved complete process shutdown reliability when some downloads are waiting on bandwidth on shutdown

did some misc listctrl update work

version 274

the help menu now has an easy on/off check entry for the darkmode colourset

changing any individual colour or the entire colourset will now immediately refresh almost all custom-coloured controls with the new colour

added a BUGFIX option to options->gui to permanently fix all discord file drag-and-drop events (as long as they contain <= 10 files and total < 50MB)

you can now set specific 'open externally' launch paths on a per-mime basis under options->files and trash

improved error reporting on a bad file launch

improved the network engine to recover from and reattempt in-progress response read errors (previously reported as ReadTimeout)

fixed the 'scroll to focused thumbnail' calculations on key events (when hitting up/down arrow key on thumbnail grid, the page wasn't scrolling correctly as needed)

known urls no longer display with the scheme (http or https) in the media viewer top-right summary

known urls in the media viewer top-right summary now tooltip their full url

wrote a new button for editing tag import options. it has a good summary tooltip. it is only in the manage import folder dialog atm, but I will replicate it across the rest of the program in the coming weeks

in some situations, the file import status window will list some timestamp note info for 'already in db' and 'deleted' statuses. see if you like it

the new listctrl will generally give its data to other consumers in ui sorted order (this fixes some stuff like 'copy sources' in the file import status window, which was copying them in random order)

manage tag parents now uses the new listctrl (and is hence now safely sortable)

manage tag sibings now uses the new listctrl (and is hence now safely sortable)

some behind-the-scenes of manage parents/siblings is a bit neater

improved some thumbnail internal media structures to fetch specific media based on hash much much faster, particularly for pages with 10,000+ thumbs. this should speed up large imports and other content update events that can result in thumb redraws

fixed an issue where dismissing a popup message could spawn the entire result of the queue, ignoring the 'show 10 max' rule

completely finished the menu rewrite! all menus now work on the new system

deleted a ton of old and now obselete menu event processing code

some boot/shutdown terminal printing should be more reliable

misc refactoring

misc cleanup

version 273

converted the gui colour options to the new options system

added a 'darkmode' colour set and extended the colour options page to support it!

added a secret discord-compatible drag and drop mode. start a file transfer drag and drop with ctrl held down to initiate. keep ctrl held down, as this is secretly a 'move' drag and drop, and you want to keep it a 'copy' one. it should work for discord dnd, at least to the web version. I have an idea on how to do this better, but feedback on this would be appreciated.

converted 'import options - files' collapsible panel to a button that launches a dialog. this reduces total gui object count when many importers are open and reduces overall ui lag and limits. the button shows a summary of its current options as its tooltip. furthermore, all of this sometimes confusing nomenclature is now uniformly presented as 'file import options'

added a BUGFIX option to gui options page to force hover windows to display at all times

improved last week's gelbooru 404 fix--page urls generated pre-v272 should now be more reliable

in a further attempt to improve workspace support, popup messages will no longer update in any way if the mouse is not on the same display as the main gui

fixed an issue where dupe import urls could be queued up in a file import queue if they came in the same batch. this was essentially harmless but lead to some mixed x/y progress counts and row indices

fixed an issue where url drops were not filling in the url entry when a new raw url page had to be created on a currently selected page of pages. this may have affected a couple of other page spawning situations

may have fixed an issue with the mouse shortcut-setting button, where for some users scroll events were not registering

manage subs dialog now has the new help button up top

manage subs dialog's sub verbs are now wrapping into a single menu button that I can expand more easily in future

sped up thumbnail access a bit on certain well-defragged hard disk drives

made thumbnail fading a little smoother in some situations

rendered images that aren't otherwise pushing any memory limits will now stay cached for less time (10 mins) while thumbnails will hang around for longer (24 hours)--this was previously 20 mins for both. you should now see less thumbnail page 'refreshing' after returning to the client after inactivity

sketched out basic classes for login and domain engines

sketched out basic class for url matching

sketched out basic classes for matching and transforming strings in certain login/domain situations

updated the manage export folders dialog's listctrl to the new control

updated the last media viewer menu code to the new system

removed some old proxy code that was sometimes intruding on the new networking engine. proper proxy support will come in a later version

fixed some test code

misc refactoring

version 272

finished moving the last misc network consumers to the new networking engine

greatly simplified and harmonised some of the new network job response processing

the gelbooru 404 issue should be fixed!

added a parsing patch for rule34hentai.net's mp4 links, which was causing 'link not found' failures on the hydrus parsing end. please tell your R34 subs to retry their fails

wrote a patch (actually updated an existing patch that had fallen to bitrot) for the danbooru booru (and extended it to anything else with 'Running Danbooru' in its html) to try to fetch the full-size image if one exists. this may not last long, so I do not recommend regular users rely on it, but feel free to play with it until I have a better fix in the new downloader engine

you can now add 'thread watcher' to the default import tag options. it has 'filename' and explicit tags options. unlike gallery downloaders, thread watchers will not inherit from the 'default' import tag options entry--you need to set the specific entry

added regular number key support to the new shortcut system

fixed the borked splash screen drag coordinate calculation

heavy session loads should be more polite to CPUs and window managers

pages should generally spawn with their proper name (rather than a flicker of 'page') no matter their spawning context

fixed osx page tab drag-and-drop coordinate calculations

figured a way to make os x page tab drag-and-drop start on a single left-mouse-down event

in an effort to improve popup display on systems with virtual desktops, the popup message parent window will now not go from hidden to shown while the mouse is on a display screen other than the main gui

fixed a wacky bug where clicking on a page tab, then on its sort dropdown, then on a different tab, would invoke the Deep Ones to commence a page tab drag and drop event

fixed an index issue when unclosing some pages in certain orders

increased pre-processing disk cache population time

tweaked regular memory maintenance disk cache population timings

fixed shortcut-driven tag petition events when the tag has a sibling (e.g. if you have sibling pair (a->b) and a shortcut (key->flip a), hitting 'key' will now correctly flip a (appearing as b) on and off in the media viewer)

fixed some ugly layout sizing calculations on the path tagging dialog

updated the path tagging listctrl to the new control

updated the quick namespaces listctrl to the new control

updated the regexbutton's menu to the new menu system

fixed some typo errors with the regexbutton's entries!

removed the old gallery downloader unit tests, as these operated on the old network and downloader engine--new, granular tests will be reintroduced for the new parsing components as they are written

cleaned up some shortcut key code

misc refactoring

misc improvements

version 271

fixed an issue where the notebook 'motion' event was being consumed, disallowing the regular OS 'highlighting' on tab mouseover

fixed page tab drag and drop crash for Linux

fixed page tab drag and drop coordinate calculation for OS X

page tab drag and drop should now work for all platforms

new 'page of pages' pages now start with a blank file search page

fixed several issues with row display in the new listctrl

added a # index column to the file import status window, so you can resort the 'to be imported' order reliably

fixed an issue where the 'delete_file' application command was not firing on an appropriately mapped shortcut

fixed an issue where automatic bandwidth override would only fire if its page was in view. it now occurs at all times

fixed an issue where the file import status frame, as launched from manage subscriptions, would sometimes be non-responsive (typically for Linux)

fixed the 'pattern shortcuts' button on the export files dialog, which was misbehaving for certain choices for some window managers

fixed an unusual error where middle-clicking to create a new page while a menu was open would lock the application from receiving new key/mouse events until program focus was lost, wew lad

updated some old network code to the new engine

added some delay to network connection error reattempts so that instant fails due to local network disconnect (e.g. while computer is waking from sleep) do not consume all the reattempts too quickly

greatly simplified low-time-delta bandwidth testing and reporting, improving essential accuracy and overall feel

gallery download errors will now print additional info to the log--let's see if we can gather more info on the gelbooru 404s

wrote a patch for users of 2.X opencv who were running from source, who were unable to boot

due to hardware failure, had to make a new Linux build computer. some libraries are updated, but build _should_ be the same--let me know if not

updated to opencv 3.3 for all releases

misc old menu code updates

version 270

added page tab drag and drop (windows only, have to iron out critical bugs for linux/os x).

dragging onto the middle of a normal tab will put the source tab there

dragging onto the edge of a tab will try to insert the tab there

dragging onto the middle of a page of pages tab will insert the tab onto the end of that page's list

dragging a page up a notebook level will work ok

dragging a notebook into itself will do nothing and not crash the client :^)

new pages that come from the main gui level (such as from pages menu or file import) now open in the deepest open notebook (previously, they would always appear in the top row)

fixed some misc page of pages bugs

fixed a bandwidth calculation that meant 1s time delta rules were working at 50% capacity (e.g. 1rq per 1s rule for domains were actually running at 1rq per 2s)

improved a bandwidth estimate calculation that was cutting out early in some situations for large time deltas

tag manager's page up/down shortcuts no longer mistakenly navigate the archive/delete filter

added a network timeout option to the 'connection' options page

reduced some hover window show/hide flicker when the media viewer is fullscreen and the OS has a taskbar that pops in for non-fullscreen windows (this mostly affected Linux)

fixed a long-time issue with yes/no dialog layout

improved some rendering of EXIF-rotated and -flipped jpegs. I expect to add more support for this (and retroactive image metadata-parsing to figure out correct reversed resolution--atm rotated images remain stretched) in the coming weeks

fixed an error when cancelling the booru-picker dialog from the page chooser dialog

if no files or all files are selected, the 'invert' select choice will no longer be shown (in this case, it redundantly does the same job as 'all' and 'none')

fixed an issue where setting namespace sort as default would persist through a client reboot

fixed an issue where force idle debug mode was not waking sleeping daemons

increased frequency of mappings processing reporting

wrote an exception and the needed maintenance code for the 'repairfilesystem' dialog to allow a proceed action if the only remaining incorrect paths are thumbnail paths--in this case, the client will create empty prefix folders and prompt the user to regen thumbnails

export filenames are now clipped to make <255 total characters in path

removed the old 'processing phase' option, which was no longer in use

removed the proxy settings from the 'connection' options page--the new engine does not use this old system, but if there is demand, a more flexible system will return

misc image rendering pipeline updates

misc improvements

version 269

nested pages now supported!

moved all page management (session load/save, new/close page, page navigation, page name maintenance, etc...) code from the main gui to the new PagesNotebook object

expanded the session object to hold nested page information

added a 'page of pages' page to the 'special' new page entry!

numerous other gui-notebook page-related event fixes and improvements

figured out cross-platform menu and other mouse event support for nested notebooks, but there may still be holes--please let me know if your new pages ever appear in the wrong tier!

ways to move pages up and down rows will come in the coming weeks!

main gui status bar now shows total bandwidth this session as well as current speed

added a cog button to network job controls that allow for manual- and auto-bandwidth-override

added a 'blocked?' column to the review bandwidth panel to quickly see which network contexts are currently available to do work

added a button to reset 'default' and 'global' bandwidth rules to review bandwidth panel

merged the network request start test and consumption code into one transaction, stopping some accidental overconsumption when the engine was under heavy load

did some logic work to make sure unusual network context rule/usage situations are visible on the review bandwidth panel for editing

network jobs now report more information as they get ready to start, including while they are held up waiting for a download slot

added a simple 'network profile mode' to the debug menu that atm just prints a summary of new jobs

fixed an error that could sometimes be a crash when 'review services' was opened while no pages were open in the main gui

fixed the pixiv login test button with a hacky workaround--I will make extensive proper login testing gui when I move to the login engine

fixed an issue in the youtube downloader

if the bandwidth or session managers are missing on boot, empty defaults will be created in their stead

bumped the max page limit up to 150--I expect to increase it more in the coming months as I rejigger how some gui stuff is laid out

renamed 'sort by age' to 'time imported'

fixed and improved some test code

cleaned up some shutdown code

should have fixed a rare unicode conversion issue when printing to log

misc improvements

version 268

split the sort dropdown into two, splitting the sort type and sort order

file sorting works more intelligently behind the scenes in several ways

added 'sort by number of tags' to sort options

session pages now remember their sort status!

sessions also more reliably remember their actual thumbnail order for all pages (typically, this matters for importers atm)

network job controls now report an estimate of how long they will have to wait for bandwidth

subscriptions will now show some 'no more bandwidth to download files' text if they have to stop because of bandwidth rules

subscription 'should I start/continue' testing now has a little padding to forestall some potential unexpected long delays in operation due to edge cases

the edit subscription dialog now uses the new file import status control

fixed an issue with hentai foundry filters not applying (they added some categories since the downloader last worked, breaking the POST form), hence hiding most non-vanilla results

finished the 'database->migrate database' dialog and its help and removed the 'under construction' labels. the new help page is also now linked from the standard index.html

fixed an issue where pages would sometimes never initialise (due to being queued up after an infinite job like the network engine's mainloop!)

improved how all long-job threads are spawned

improved some more thread scheduling logic that meant some long-term jobs could be stacked undesirably

expanded how the new listctrl updates and deletes its data

reduced flicker on the new listctrl update events

the file import status window now uses the new listctrl

the manage subscriptions dialog now uses the new listctrl

collections now track their tags more accurately

moved the 'delete original files after success' checkbox up to reduce misclicks on ok'ing the file import dialog and added a bit of red warning text whenever it is on

sped up some behind-the-scenes content processing for large pages

improved how the program cleans some things up during exit

the issue with some clients not clearing their exit splash screen until a mouseover event should be fixed

at 120 open pages, the client will inform the user about the approaching max page limit

by default, import folders no longer delete anything. default is to ignore original files in all cases

deleted some old unusued code

misc cleanup

misc improvements

version 267

drag-and-dropping a 4chan or 8chan url onto the client will now automatically open a thread watcher for that url

fixed an issue where web domain or subscription network contexts that included unicode characters in their context data were unable to serialise and save to the db, causing error spam when the bandwidth manager attempted to save itself

subscriptions will now show a network job control in their popup as they do network work

subscriptions will cancel more reliably during gallery parse

subscriptions now have some 'delay' logic that will stop them sometimes restarting as soon as they are cancelled or otherwise have to stop mid-work

subscriptions will now tolerate up to five 'already in cache' urls per page parse until it considers the page 'already seen'. this is to catch the odd additional late insert and avoid the problem of a page updating and shuffling everything up one as the subscription walks through pages

maintenance modal popup messages should no longer appear if their jobs are very quick

queued up modal popup messages will no longer flicker their dialogs on a de-minimise

should have fixed an issue where modal popup messages could sometimes error out on a close attempt, locking the client's whole gui and requiring a force quit

fixed an issue where the modal popup message dialog was OKing on a close attempt of a non-cancellable job, despite presenting user text indicating otherwise

did a bunch of data work on pages

page tab names will now be clipped to 20 characters by default

pages now show (num_files) after their name by default, although you can set this for only import pages or turn it off completely

options for these new behaviours are in options->gui--and page names will update immediately on dialog ok

subscription and import folder 'show files' events (and a couple of other misc occasions) will now launch their page tab with the sub/folder name rather than a flat 'files'

wrote a new listctrl class with custom sort code that does not suffer from the 'crash when sorted while many pages open' bug. it also handles data in a simpler way for hydrus

review bandwidth listctrl now uses this new class and should now not crash your client. I will replace all the other listctrls with the new class over the coming weeks

temporarily, no old listctrls will auto-sort themselves (as this causes a crash for many users)! if you do not suffer from the crash, please sort them yourselves for now

reduced some large-scale gui import lag:

reduced content processing CPU load on clients with many thumbnails open

massively reduced content processing CPU load on non-current pages

offloaded newly-imported file thumbnail detection and generation to a non-gui thread

migrate database dialog now lets you move the whole database and all portable locations, which requires a client shutdown

the raw url downloader will no longer have a problem with pasted url lists that include empty newlines

all the downloaders/importers now sleep on cleverer event objects so they will burn less idle cpu and wake as soon as they have new work to do

fixed and otherwise improved some gallery downloader timing logic

hdd import pages now use the file import status control

the thread watcher's controls now try to wrap in a single sizer. it uses less space, but might sperg out, let me know if it is a problem. I'll replace the whole watcher timing system in the new engine anyway

fixed a gelbooru (and possibly others) booru parsing bug that meant half of the pages in the gallery walk were being skipped (e.g. 'mogudan' was producing ~350 files when there were actually 690 in the list)

fixed some dialog panel layout scrollbar-cutoff in sevaral Linux places and perhaps elsewhere

cleaned debug menu a little and added a save 'last session' entry

fixed an issue where some kinds of media would error on notification of new url association

clients will now save small transaction progress reliably within ten seconds no matter how idle they are

deleted the old 'gui capitalisation' option--I never got around to expanding it beyond a handful of menu labels, it was always too much to work on for too little reward

improved support for certain broken videos--these will import ok, but full rendering might be borked to different degrees, so let me know how it goes.

cleared out some old content processing code

cleared out some old unused db data

removed all old 'waiting politely' download settings and gui code

misc prep work for wx update

misc gui code refactoring

misc downloader cleanup and timing tweaks

misc improvements

version 266

converted gallery downloaders to the new network engine

greatly simplified how gallery downloaders report network activity and converted them to show the new network job control as well

subscriptions also work on the new engine but will not show network gui yet

hacked hentai foundry and pixiv login to use the new network engine

successful logins to hf or pixiv now print to the log

the new network engine now clears temporary session cookies after 90 mins of inactivity

gallery downloaders and subscriptions now use the new 'downloader instance' and subscription bandwidth rules. by default, this means downloaders will do small bursts every five minutes and that subscriptions will do at most 256MB per day

subscriptions' bandwidth use is now listed by name in the review bandwidth panel

subscriptions use a new bandwidth test to determine if they should start or continue based on current bandwidth limits. it should mean subs do a good bit of work and then stop when they are supposed to without ever waiting on bandwidth more than 30s or so

thread watcher and page of images now ignore bandwidth limits when doing their 'page' fetching part

gallery page fetching will ignore bandwidth rules (in order to stop gallery walk desyncs from having to wait a long time)--it will fetch one page per five seconds

the thread watcher and page of images importers now work on their files and page-checking simultaneously--also, the page of images will process its page queue at any time, not only when the current file queue is finished

the gallery downloader page now works on its files and gallery page parsing simultaneously!

wrote a 'file import status' control to better wrap up the import summary, progress gauge, and file import status button into one panel

thread watchers now use the new file status control

'page of images' downloaders now use the new file status control

the gallery downloaders now use the new file status control

all network jobs will now retry up to four times on the BadStatusLine ConnectionError, which seems to be a TLS (https) negotiation timeout/remote termination

all requests on the new network engine will now timeout after ten seconds

they will also retry on generic timeout errors

popup messages can now be shown in 'modal' mode as a dialog that prohibits interaction with the rest of the client.

these will not boot while the client is minimised

database maintenance routines will now all publish messages like this

'migrate database' panel now publishes a modal message when it does a file rebalance

rewrote the controller-side pubsub pipeline to respond faster and consume fewer program resources, particularly for the client

import pages now update themselves in a less spammy way behind the scenes, meaning more active pages can be open at once without them stepping on each other and clogging things up

simplified how importers set their status information in several ways

reduced a swath of pubsub spam related to content updates

improved how spammy small jobs are written to the profile log

reduced text flicker on all download pages

more misc pubsub improvements

reduced some gui update/refresh spam on hidden pages

cleaned up a bunch of database->gui message reporting and cleanup code

added close other/left/right pages to tab right-click menu

the top-right media hover window will no longer refit-flicker on a ratings change

wrote a new panel wrapper for listctrls that handles the underlying row of buttons in a neater way and automatically disables them if they are nullipotent (mostly, this means greying out 'delete' buttons when nothing is selected)

several listctrls use this new panel

when it is not strictly necessary, videos that are >30MB will no longer use the CPU-expensive manual frame count parsing

a problem where newly reloaded thread watchers could sometimes stick in a 1/1 initialisation state _should_ be fixed

fixed gelbooru url parsing (they stopped using the janky redirect.php urls)

fixed an issue that meant ipfs pin was erroring when trying to show gui-side

'page of images' downloaders now protest if they are told to close while working

some small layout and status text fixes

adminside petition processing now has a 'flip selected' box to flip checked status of all selected contents

adminside petition processing contents chechklistbox now supports multiple selection

improved the reliability of some shutdown code

cleared out some old unused code

misc new control cleanup

misc fixes

more misc fixes

version 265

the bandwidth engine now recognises individual thread watcher threads as a network context that can inherit default bandwidth rules

tweaked default bandwidth rules and reset existing rules to this new default

review all bandwidth frame now has a time delta button to choose how the network contexts are filtered

review all bandwidth frame now updates itself every 20 seconds or so

review all bandwidth frame now has a 'delete history' button

review all bandwidth frame now shows if services have specific rules

review all bandwidth frame now has an 'edit default rules' button that lets you select and set rules for default network contexts

review network context bandwidth frame now has a bar chart to show historical usage!

bar chart is optional based on matplotlib availability

review network context bandwidth frame now lists current bandwidth rules and current usage. it says whether these are default or specific rules

review network context bandwidth frame now has a button to edit/clear specific rules

rows of bandwidth rules and current usage, where presented in ui, are now ordered in ascending time delta

misc bandwidth code improvements

client file imports are now bundled into their own job object that generates cpu-expensive file metadata outside of the main file and database locks. file imports are now much less laggy and should generally block the feel of the ui much less

removed the database 'rebalance files' menu entry

removed the 'client files location' page from options

db client_files rebalance will no longer occur in idle or shutdown time

(this stuff is now handled in the migrate database dialog)

'migrate database' now uses a dialog, meaning you cannot interact with the rest of the program while it is open

migrate database now has file location editing verbs--add, remove, +/- weight, rebalance_now. thumbnail location and portable db migration will be added next week

flushed out the backup guide in the getting started help, including to reflect the new internal process

the client now saves the 'last session' gui session before running a database backup

the shutdown maintenance yes/no dialog will now auto-no after 15 seconds

gave status bar tabs a bit more space for their text (some window managers were cutting them off)

tumblr api lookups are now https

tumblr files uploaded pre-2013 will no longer receive the 68. subdomain stripping, as they are not supported at the media.tumblr.com domain (much like 'raw' urls)

pages will now not 'start' their download queues or thread checkers or whatever data checking loops they have until their initial media results are loaded

key events started from an autocomplete entry but consumed by a higher window (typically F5 or F9/ctrl+t for refresh or new page at the main gui level) will no longer be duplicated

fixed a shutdown issue with network job controls that could break a clean shutdown in some circumstances

if the user attempts to create more than 128 pages, the client will now instead complain with a popup message. Due to OS-based gui handle limits, more than this many pages increasingly risks a crash

if the client has more than 128 pages both open and waiting in the undo menu, it will destroy the 'closed' ones

version 264

converted the page of images downloader to the new network engine

converted the thread watcher downloader to the new network engine

rejiggered page of images and thread watcher management panels to better separate the two loops they work on

rejiggered some more downloader layout stuff in prep for a new control to better display url cache and overall import status--downloader-global pause button is now always at the top, for instance

removed 'waiting politely' indicator from download pages that use the new network engine

network job controls now have a simple border

network job controls will show current bytes progress and speed in a more intelligent and useful way--no longer spamming the initial 0KB/s, for instance, and flashing more useful summary information when downloading many small files

network job controls now display their current bytes progress and speed on a separate right-aligned text beside the main status text (to stop the speed text jumping around so much)

network job controls will not show the current download speed until some bytes are actually read

wrote a first version of 'services->review bandwidth' review frame, which lists all network contexts with data in the past month

wrote an independant and live-updating review frame for indivdual network contexts, launched from the review bandwidth frame, which shows current and historical bandwidth use for that context

the main gui status bar now reports current bandwidth usage! updated once a second

database->backup process now has some nicer gui, better and more reliable workflow, improved file copy notification, and now correctly obeys a popup-cancel command

started the new 'migrate database' gui under the database menu. it currently shows live usage and will gain some verb buttons to atomically alter things hopefully next week

moved from a strict begin/commit db model to a save/release system that only commits changes to disk at most every ten seconds or so. small 'write' transactions will now often have no disk-sync overhead and so are super fast, particularly when they come in a large batch.

simplified database connection, disconnection, begin, commit, and rollback code significantly

the url table in the client database is now non-unique (i.e. multiple files can have the same url). this may lead to undesired 'already in db' or 'deleted' statuses in some downloaders as they check for url status before starting their downloads. if this change turns out to a be a big pain in the future for pixiv manga or whatever, it may be revisited.

fixed a bug in deviant art parser that sometimes meant nsfw urls could not be found

fixed an issue where progress gauges could sometimes get stuck in a pulsing state

fixed a bug with the new optimised result building which sometimes occured when all the files in a batch had common ipfs service membership

improved support for unusual videos

improved accuracy of duration calculation for unusual videos

improved database optimisation after initial and ongoing repository processing

a 'wake subscriptions' event now occurs even on 'manage subscriptions' cancel (meaning temporarily-dialog-paused subs will always continue after the dialog is closed)

added a careful manual commit to stop the possibility of the definition-desync some users noticed after a power-loss during a big repository sync commit

removed some redundant old db compatibility code

permitted sqlite installations that allow in-memory temporary tables to use them (but not for vacuums, which tend to be a bit big for this stuff)

cleaned up some redundant timer id code

misc improvements

misc refactoring

version 263

greatly improved how gui sessions are loaded--now the page tabs are loaded instantly, but the thumbnails are loaded in the background. session loaded should be significantly less laggy and buggy

the issue of pages sometimes initially sizing at zero size (which could be caused by minimising the client while a session load was ongoing) should hence be better/fixed completely!

gui sessions will now load their files in the exact order in which they were saved--behaviour that I think was not previously always reliable

more general network code work and polishing

added and improved unit tests for network code

improved how short-time-delta data bandwidth is reported

improved how short-time-delta data bandwidth is tested

wrote a networkjobcontrol to display and control the new network job object

tumblr parser now produces 68.-less urls

tumblr parser now produces https urls

cleaned up tumblr parser a little

url caches will clip existing tumblr urls of the 68.-subdomain and convert to the new raw format, clearing out dupes along the way

url caches will convert existing tumblr and pixiv urls to https, clearing out dupes along the way

the pixiv parser now deals with missing creator/title tags without errors

extended the specific file domain tag cache to also store deleted mappings, resulting in much faster request building for clients with large numbers of deleted mappings

improved some downloader page queue text display timing

added support for more types of mp4 file

improved how some memory maintenance calls work

improved how hydrus datacaches track their recently-used-data fifo list

pages now regularly clear out spare thumbnail canvas bmps

pages now regularly clear out cached thumbnail canvas bmps when they are not the currently viewed page

import caches, when asked for url membership, will now test both the http and https versions the url

maybe improved how 'open in file browser' works in windows

fixed the 'recount video frames' advanced thumbnail menu entry, which wasn't working with the new ffmpeg wrapper

moved some bloaty master hash data out of client.db and into client.master.db

pubsub profile will no longer profile 'message', as it just makes for spam

reduced some serverside pubsub spam

reduced some significant clientside pubsub spam that I think was smashing the gui event loop at inconvenient moments

improved some client shutdown object sync code

fixed an issue where some duplicate maintenance popups would not clear themselves up properly if interrupted mid-job

cleaned up some http-https conversion and comparison code

fixed some status-setting code that meant thumbnail pages were sometimes setting status after they were replaced and scheduled for deletion

misc improvements

version 262

added apng support!

sessions will now append from the default insertion index (e.g. right of current page)

sessions will now load in the correct order if the default insertion index is to the left of the current page!

fixed an issue with custom 'favorites' gelbooru parsers

manage parents/siblings dialogs now load their larger datasets on a thread, after initialising

sped up how some default thumbnails are generated on client boot

sped up how resized thumbnails are generated

refactored ffmpeg video parsing to be simpler and easier to maintain

improved some ffmpeg parsing to be much faster

ffmpeg can now parse video that reports no duration

fixed some more bad video framerate parsing

the 4chan/8chan thread watcher will now always generate https urls (even if you enter an http thread url)

pixiv will now produce https urls

the known url file status system will now check both http and https versions of an url it is given

refactored some core elements to simplify common controller requests

started on some database migration help and ui--should be finished next week

flushed out network engine a little

more refactoring and simplification of new network engine object coupling

flushed out bandwidth manager

misc logic tweaks in bandwidth management

wrote a network session manager

wrote a new class for identifying all future 'network contexts'

employed NetworkContext everywhere in the new engine, massively simplifying several things

wrote the first batch of tests for my new network engine and its network jobs

wrote tests for the bandwidth rules object, including covering the new short time delta support

improved reliability and speed of bandwidth unit tests

wrote tests for the new domain-based bandwidth manager

misc improvements

more misc improvements

version 261

wrote a new manage dialog for urls

added 'manage known urls' to media right-click menus!

double-left-clicking on any video animation will 'open externally'!

added an option to options->gui to set where new pages will appear by default--either far left/right or left/right of current page

the tumblr parser now produces '_raw' urls when the post was posted in 2013 or later

created a new 'number' subtag cache that will be populated on update

created a new 'tag as number' system predicate that can search for, say, all 'page:' tags > 200

bandwidth management now tracks requests and num_bytes more sensibly

bandwidth tracking objects can now better handle bandwidth usage and rule application in short intervals (i.e. sub-5-second) (however, the current networking engine cannot yet use this information accurately)

wrote unit test for bandwidth tracker, including for the new short interval timing

fleshed out the new network engine

fleshed out the new network job

thread watchers can now have a time delta that includes days (giving a max check period of 360 days, wew)

adminside mapping petition processing now has adaptive max total petition weight--lower file count range petitions will have much higher max total permitted weight

added a new 'callto' debug reporting mode that reports on current thread pool jobs

improved calltothread pre-spawning checks to reduce outside chance of deadlock in busy periods

the advanced review services repository panel buttons are now hidden unless in advanced mode

eliminated some animation buffer looping redundancy

fixed a little animation next-frame prediction code

fixed up some '1 minutes'-type time_delta->string conversion

fixed up label on time delta control button

fixed some shutdown thread interactions

fixed an issue where sometimes empty tags could be entered into the manage tags panel

added some pydeadobject error handling during client shutdown

refactored some db multi-tag->file search code, cleaned up wildcard searching

misc string cleanup

misc dialog cleanup

version 260

fixed video parsing when the video metadata includes random non-utf-friendly garbage

fixed video parsing when ffmpeg reports no fps at all

improved video frame counting accuracy

thumbnail waterfall process now adapts to the current speed of the cache and can hence use significantly less overhead

the thumbnail fading process now adapts to the current rendering speed, delivering smooth fade when CPU is available but otherwise skipping frames and more reliably filling in thumbnails within a quarter of a second

canvases now draw their thumbnails with slightly less overhead

increased database synchronous pragma to FULL to better safeguard against power loss during multiple-db commit--we'll see if it slows things down too much, and maybe add an option about it

cleaned up and improved some client gui pubsubs

made pubsub profile mode far less popup-spammy

if a profile takes less than 20ms, it now won't be fully printed to the log

tweaked some more server object maintenance code

added 'changelog' link to help menu

updated lz4 library and fixed some old deprecated calls

misc serverside pubsub cleanup

misc fixes

misc refactoring

misc code cleaning

version 259

planned out new networking engine and started the principal objects

renamed the 'exact match' duplicate status to 'same quality', to reduce confusion on what is appropriate for this status

the duplicate filter, on hitting the delete key, now offers the option of deleting both files

the duplicate system now combines duplicate status setting and the consequent batch of content updates into the same database transaction, speeding things up

the duplicate system now batches multiple duplicate status setting into a single transaction, massively speeding up large filters or thumbnail status set actions

misc duplicate help tweaks

you can now edit the default duplicate merge options from the new thumbnail duplicate menu

the duplicates page's jobs are less demanding on gui time and take better breaks if something else happens

renamed the new dupe system predicate to 'system:num duplicate relationships' to clarify what it searches

for normal queries, current and pending mappings are now fetched from a faster mappings cache. you should see faster result building across the board, particularly on fresh boots or otherwise slow-disk-access systems

added a prototype 'advanced mode' (defaulting to off, so experienced users will want to turn it on) under the help menu that will enable menu items that are often not helpful to new users. I will add more things to this in future, suggestions welcome

the new thumbnail menu dupe relationship set stuff is now considered advanced

thumbnail menu find similar files is now considered advanced

the thumbnail menu copy hash entries are now considered advanced

advanced content update buttons (on manage tags and review services) are now considered advanced

added an advanced mode 'open file location' entry to the thumbnail share menu, which will open the file in your OS's file explorer (not available on Linux)

added an advanced mode 'correct video frame count' thumbnail menu entry that will force-apply last week's more accurate video frame counter to correct and videos that render too fast and then cut off

fixed many entries on the media viewer menus, which were being blocked by an over-eager 'can continue' test and hence silently failing

fixed the youtube downloader on Linux and OS X--both now use youtube-dl

fixed an issue where the GetLighterDarkerColour function was producing very bright alternates to very dark colours (meaning dark dupe filters were having their bright background text rendered unreadable)

improved video frame number parsing accuracy

improved the accurate version of video frame number parsing accuracy, particularly for longer videos

the network engine now reports 5xx http status codes as ServerException to better contextualise to the user what went wrong

adminside mapping petitions are now sub-ordered by tag

adminside sibling/parent petitions are now ordered by the 'older' tag and sub-ordered by the 'newer' tag

censorship taglists are now roughly sorted

reduced default shutdown work max time to five minutes

improved how subprocesses are started

misc cleanup

misc refactoring

version 258

added a duplicate entry to the thumbnail right-click menu

the new duplicate menu will now attempt to fetch known duplicate counts for the focused file for the current file domain (if the db is locked, it will say so rather than block)

the new duplicate menu's counts in the thumbnail menu can be clicked to show those files in a new page

the new duplicate menu allows you to set the four main dupe statuses (with default merge options), or a customised verson of the same, or set unknown/potential, or delete dupe status completely, for all the possible pairs in the current selection

added tentative support for variable framerate files, manually counting up their frames and displaying them with an averaged constant framerate in the client renderer. this now permits a number of webms that were previously 'mime unsupported'. let me know how it works for you!

cleaned some server locking code, hopefully fixing the ssl handshake issue (which is actually a deadlocking issue)

export file dialogs will remember the last value of 'export tags to .txt files?', and clicking that checkbox will always launch the dialog for editing (rather than clearing if there are some set)

fixed the advanced content update (as used by service-wide update), which was not able to differentiate namespaces since the recent service changes. I apologise to anyone affected by this--I will add some unit tests to make sure it doesn't happen again

reworked and cleaned some canvas event handling code

the canvas will now accept mouse wheel events even when it does not have focus

the canvas frame will no longer accept keyboard events (and hence activate fullscreen_switch) if the mouse is over a flash window

hydrus servers now respond to /robots.txt with a 'disallow all'

servers will now provide mappings petitions to admins in groups of similar size, rather than mixing petitions of weight 5000 with a hundred of weight 1

fixed the back/skip buttons in the archive/delete filter's top hover window

the file path tagging dialog's namespace and regular regexes now support parenthesis groups, and in that case will take each of those submatches instead of the wider match

hence regexes with groups will now compile in that dialog

you can now 'new page here' on the final (rightmost) page tab

fixed an issue where media-viewer-launched manage tags frames would yield focus to the main gui if they had opened a sub-dialog while they were open

the tag censorship edit panel now has a 'help here -->' label in blue text

file imports will fail properly when the final temp_path->client_files file copy operation fails (due, for instance, to the destination being on a removable medium that was recently disconnected)

fixed serverside upnp maintenance, which was silently failing early, and also updated it to the new server object code

wrote a new checklistbox dialog that handles the underlying data explicitly rather than messing around with external text->data maps

replaced all the old string checklistbox dialog instances with the new one

subscriptions will now correctly clear their 'recent error' status on a successful sync

refactored and cleaned how basic mediaresult objects store and consult simple file information

locationsmanager objects now handle inbox status and the related content update events that affect it

fixed a bug in HydrusTagArchive when attempting to figure out hash type from an existing hash

removed the obsolete servertoclientcontentupdatepackage object

misc duplicate code cleanup

misc serverside service code improvements

misc small fixes

misc cleanup

version 257

the duplicate filter will now maintain zoom on files with the same ratio

split the duplicate merge options into separate tag/rating controls--you may see some duplicate service entries, but these will be cleaned on your next shutdown

duplicate merge options now allow syncing 'archive' status

duplicate merge options now allow 'delete both files', which you may find use for in custom actions

created a tag censorship object to handle and action a rich tag censorship ruleset

tag duplicate merge options now have and use this tag censorship object to filter which tags are merged, with an initial value of 'let everything through'

wrote a tag censorship edit panel and tied it into the duplicate action edit panel so these new tag censorship objects can be edited

added an optimisation to the duplicate status setting code--if two files are better/worse, they are inherantly duplicates and so 'not dupe' and 'alternate' relationships apply to both equally and can be duplicated

fixed and culled and normalised the 'this has more tags' dupe filter statements to be more accurate and useful

added 'this has a larger filesize' type-statements to the dupe filter

created a 'system:duplicate relationships' predicate that can find files based on how many duplicate relationships of a particular type they have

cleaned up some misc duplicate filter code and added some tooltips to the top hover window dupe action buttons

added 'move to left/right end' to main gui page tab right-click menu

added 'new page' and 'new page here' to main gui page tab right-click menu

you can now right-click for a menu from empty tab space on the main gui

the main gui statusbar now updates more efficiently when under heavy refresh load

the main gui statusbar now shows db read/write/commit status and sets the current db job summary as its tooltip--if you experience persistent hangs, please hover over the statusbar and report what you see!

export tags to .txts checkbox will now default to 'all services on' when checked

fixed the thread watcher, which was accidentally disabling its text input early

'similar files' searches launched from the thumbnail menu will now default to 'my files' file domain rather than 'all local files'

downloader pages will now correctly sort their files on initialisation

refactored and generally cleaned up some collect and sort code

fixed some unlikely-but-possible collect/sort bugs

fixed some bad layout in the top-right hover window that was making it grow unreasonably tall when many urls were shown

on the different download import pages, the progress gauge that shows file download progress will now reset back to 0 as soon as the file download is complete

fixed a problem where video imports with unicode characters in their path were failing to mime-parse

improved the file import status update pipeline to better deal with large transactions (like skipping/deleting/retrying a thousand rows at once). all these big transactions should lag the gui far less

improved some misc import status cache code

made first step in a big size rewrite job that will size many elements according to local system font size rather than specific pixel values

hydrus servers now explicitly default to TLSv1.2--we'll see if that clears up some of the handshake timeout problems we have recently seen

cleaned up a bunch of possible pydeadobjecterrors when the new review services panel is closed

improved and rescheduled gelbooru redirect url purge

added a catch-and-recovery to hydrus network session initialisation, which may sometimes receive invalid data after a service deletion

added similar catches to tag parent/sibling initialisation, which apparently can be vulnerable to a similar invalid data problem

I think I cleaned up some more Linux ClientToScreen console errors

refactored and cleaned some frame size event responsibility

refactored and cleaned the panel and controls that display file import status

did a little more menu code cleanup

misc cleanup

misc fixes

version 256

the duplicate filter now loads new pairs off the gui thread. it will display 'loading pairs...' during this time

media viewers of all kinds are now more comfortable displaying no media (when this occurs, it is usually a frame or two during startup/shutdown)

the duplicate filter now responds to any media_viewer_browser navigation commands (like view_next) with a media switch action

you can now alter the duplicate filter's background lighten/darken switch intensity from its top hover window's cog icon

fixed a bug in the new dupe pair selection algorithm that was preventing pairs from being presented as groups

the duplicate filter will now speed up workflow by automatically skipping pairs when you have previously chosen to delete one of the files in the current batch

auto-skipped pairs _should_ be auto-reverse-skipped on a 'go back' action

added a |< 'go back' index navigation button to the duplicate filter top hover window

the duplicate filter now displays several 'this file has larger resolution'-type statements about the currently viewed file. it lists them on the top hover window and in the background details text underneath

the duplicate filter _roughly_ attempts to put the better file of the two first. this will always be indexed 'A'

the duplicate filter now shows done/total batch progress in its index string--not sure how clear/helpful this ultimately is, so may need to revisit

an unusual bug where Linux would spam the 'All pairs have been filtered!' duplicate filter message over and over and then crash _should_ be fixed--the filter no longer waits for that message to be OKed before closing itself

drag-and-dropping text onto the client will now a) open a url import page if none already exist and b) put the dropped text into the input box of the first open url import page (and focus it, so you can quickly hit enter)! This works when dragging text links from browsers, as well

you can now 'append' gui sessions, which will just append that session's tabs to whatever is already open--say, if you have several 'favourites' pages you want to be able to quickly load up without having to break your existing workflow

ipfs services now have a 'check daemon' button on their review services panel which will test the daemon is running and accessible and report its version

fixed the 'test address' button for ipfs services on their manage services panel

the client can now automatically download files it wants and knows are on an ipfs service

middle-click on an 'all known files' domain thumbnail will now correctly start a download (as long as a specific remote file service is known)

the multihash prefix option is reinstated on ipfs manage services panels

the gelbooru parser now discovers the correct page url to associate with its files

wrote some redirect fetching code to fix the gelbooru bad urls issue

discovered a quicker fix for the gelbooru issue--the redirect location is the garbage in the original url in base64

all downloader/subscription url caches will purge any old gelbooru 'redirect.php' urls on update

fixed an issue where 'previously deleted' gallery/thread imports were returning 'fail'

fixed a problem that was causing some redundant laggy work in adminside petition processing

thread watchers will now remember their file and tag import options through a session save even when no thread url has yet been entered

fixed an issue where media 'removed' from a media viewer view of a collection resulted in the entire collection being removed at the thumbnail level

fixed an issue where media deleted from a media viewer view of a collection resulted in the media not being correctly removed from selection tags

tag, namespace, and wildcard searches on a specific file domain (i.e. other than 'all known files') now take advantage of an optimisation in the autocomplete cache and so run significantly faster

fixed a hover window coordinate calculation issue after minimising the media viewer on some platforms

removed some 'all files failed to download' spam that could sometimes occur

misc fixes

version 255

the duplicate filter now supports shift+left-click to drag, like the archive/delete filter (this remains hardcoded for now)

if a pair in the dupe filter has the same resolution, they will now maintain zoom and pan when switching back and forth (I might increase this to work for same ratio as well, let me know how it works in the real world)

the duplicate filter will show a lighter/darker background colour as you scroll the current pair

the way lighter/darker and alternate lighter/darker colours are calculated is now centralised and should be more reliable in edge cases

improved the dupe filter pair selection algorithm--it now chooses pairs more reliably under edge case conditions and prioritises decision-groups that have high potential decision value. it should also run a little faster

increased the dupe filter batch size to 250, let's see if it causes any problems

the close button on the dupe filter's top hover window now works

fixed the duplicate filter disappearing pairs that were skipped (meaning you could not go back to revisit them)

fixed a frequent deadobject error when the dupe filter closes

the shutdown 'maintenance due' test is less sensitive to dupe search tree rebalancing, which typically only takes half a second

the archive/delete filter now uses the new shortcuts system for both keyboard and mouse input

the archive/delete filter now intercepts archive or delete commands from different sources ('media' shortcuts, top hover frame button presses) more reliably and converts them into filter actions (hence moving on to the next file)

you can now move pages one to the left or right from their menu!

on the top-right hover window (and the background underneath), ratings are now on the top, and hence will always be in the same location as you scroll through your media regardless of known urls, remote location, or inbox status

autocomplete searches for tags with apostrophes, quote marks, braces, brackets and paretheses should be more reliable

urls are now associated with files through the same 'content' pipeline as tags and ratings and so on

gui-level media is now aware of the 'new url content update' event and will update and redraw itself appropriately

fixed deviant art nsfw parsing, but it might not hold for long. proper fix here is to wait for the downloader overhaul

networking engine now uses the 'requests' module's CA .pem (which the overhaul will be moving to anyway), which should reduce the frequency of ssl verify failures (gelbooru on the relatively new 'Let's Encrypt' CA had this problem for many users)

fixed the networking engine redirect parsing for gelbooru's unusual location header. unfortunately, gelbooru is still giving mickey-mouse garbage redirect urls from its main thumbnail pages, for which in this engine there is no immediate fix

neighbouring .txt tags will now be properly cleaned and sibling-collapsed in the path tagging dialog

neighbouring .txt tags will now be properly sibling-collapsed in the import folder workflow

making a media viewer borderless fullscreen and back will now recenter the media (previously, the current drag delta was not reset, so this frequently put media off-screen)

fixed a serious issue where the media viewer could lock the client up on opening with a video if its gui options set 'remember size' to false

tag import options objects will cleanse themselves of missing services on options save/client shutdown

manage tags now defaults to cross-referencing 'my files' on the 'local tags' domain, where 'remote' (i.e. deleted) files' tags are not useful

reduced memory use when importing large pngs with transparency

improved adminside petition processing gui reporting

servers will now cap the size of their mapping petitions so as not to ovewhelm the admin processing them (they now won't be both >20 tags and >1000 total row weight)

the media viewer's manage tags frame now listens for content updates from outside, so if you alter an in-view file's tags (such as with a shortcut key), the manage tags dialog will update as it happens

moved a number of the buttons on the top hover windows to the new unified internal command engine (which the new shortcut system also uses)

disk cache maintenance uses fewer resources but now occurs in the foreground (and should hence more reliably maintain the cache)

critical repository service id lookup errors will now automatically reset the repository's processing cache and better inform the user of what has happened. if you see this, please let me know the details and how this error fired in the real world

updated and reinstated the ipfs service panel in review services

updated some ipfs service code for the new service system

misc refactoring and cleanup

more cleanup and deletion of redundant old pubsub command code

updated ubuntu build machine to 17.04 and opencv 3.2

version 254

the duplicate processing page is now on the new page picker, under the new menu 'special'

improved the dupe pair selection algorithm--it should be faster for everyone

added help icon to dupe processing page--it can launch of a couple of message boxes with simple help, or you can open some html help

wrote some dupe processing html help!

the dupe processing page now refreshes its numbers when the dupe filter closes

the archive, inbox, delete, and undelete buttons on the duplicate filter's top hover frame now work

delete and shift+delete keys now work to delete/undelete a file in the duplicate filter

the mouse cursor now hides/shows on halt and new movement on the duplicate filter

you can now set 'content' shortcut actions (adding tags or ratings) to shortcuts in the 'media' reserved shortcut set (which are always on in media contexts), although they don't yet work in the thumbnail view yet

added a help button to the manage shortcuts panel

the edit shortcuts dialog will now now allow you to name a custom shortcut set to one of the reserved names and will explain the problem in a message box

import pages update more efficiently during periods of busy cpu

import pages will use less idle cpu time generally

import pages will adaptively use less cpu time when they are in the undo deletion queue or have no files to import

the page of images downloader will spam a little less idle time

the canvas background details and the top-right hover window will now only show 10 urls max

added a help button to the tag import options panel to better explain the namespace selection and explicit tags

added 'gui report mode' to the help menu, which will report key and mouse shortcut events and and matched commands

the adminside petition panel now sorts multiple petitions by number of files descending

misc improvements

misc help cleanup

misc fixes

version 253

created a new object to hold tag and rating merging and 'worse file' deletion options

wrote a dialog to edit this new object

established some simple defaults for this object for different duplicate status actions

the cog icon on the duplicates filter now lets you edit these defaults

the 'custom' duplicate filter action now works--first by asking you what status you want to set, and then by throwing up the new merge options dialog to tune it to whatever you like

wrote comprehensive unit tests for the new object

fixed the super slow dupe filter launch time problem

added a 'known urls' submenu to thumbnail and browser canvas right-click menu that lists all known urls for a file with an option to launch or copy the url

added known urls' hosts to the top-right canvas details background, just below where known file repos are listed

added the same known urls' hosts as clickable hyperlinks to the ratings hover window that pops over that top-right area

added 'delete from deleted files' action to the local tags's service-wide update panel. it will limit the deletion to mappings that are currently on files that have been previously completely physically deleted from the program

fixed namespace filtering on service-wide update panel

the hentai foundry downloader broke, so the update code will pause all hf subs. the solution is not trivial (it is part of the downloader overhaul), but I will try to fix it soon

debuted a new question-mark help button to better explain .txt tag importing on the manual import tagging dialog and the manage import folders dialog

fixed a small potential error due to bad parsing in the 'page of images' downloader

fixed a typo bug that stopped the 'delete shortcuts set' action working in the manage shortcuts dialog

I may have fixed an issue where the server was sometimes not shutting cleanly with a keyboardinterrupt

fixed the media embed button not reliably updating its thumbnail

fixed an issue where a dummy animation bar was displaying on embed buttons that showed static images that included transparency

the serious db missing tag and hash states will now not throw an error but will inform/spam the user (and hence not prohibit a boot)

attempting to open a second manage tags frame from the media viewer will now instead put the focus on the first (previously, multiple manage tags frames could be made)

misc db code cleanup that should result in faster result building in certain situations

misc improvements

misc layout fixes

version 252

the duplicate filter now processes pairs in batches and hence supports 'back' actions to revisit decisions. you will be prompted every fifty or so pairs to commit and checkpoint your progress

the duplicate filter now presents related pairs together, rather than picking at random

fixed a bug in duplicate filter shortcuts initialisation

simplified duplicate filter default shortcuts (these will be overwritten on update) to only use simple left- and right-click for 'this is better' and 'alternates', since those are by far the most common actions. middle-click now goes back, like the archive/delete filter

converted old 'main shortcuts' system to the new shortcuts system, also splitting it up into 'media', 'main_gui', 'media_viewer', and 'media_viewer_browser' constituent parts that will be applied in different contexts

because the change is so significant, all clients will have their old 'options' shortcuts reset to the new default--I expect to expand shortcuts further in the next few weeks, so this default-overwriting will likely happen again, so you will likely wish to wait before recustomising your basic shortcuts

completely eliminated the old main shortcuts system--all references now bodge with the new system to varying neatness

as the old main shortcuts system no longer exists, the 'shortcuts' page on file->options is gone--all shortcuts are now managed through file->shortcuts, which is a completely revamped version of the custom filter shortcuts editing dialog

selecting shortcut commands is significantly simpler for the reserved shortcut sets

all media viewers with hover windows now have a 'keyboard' shortcuts icon button on their top hover window--it links to the manage shortcuts dialog as well as the current active custom shortcut sets and default custom shortcut sets

as the above system supercedes the old custom filter system, custom filters are completely removed from the program! your existing custom shortcut sets will survive, but you probably want to purge them of all the redundant junk they still have

all the media canvases use the new shortcuts system

the canvas frame uses the new shortcuts system

the thumbnails canvas uses the new shortcuts system

the main gui uses the new shortcuts system

some other misc places use the new shortcuts system

most simple shortcut command actions have been renamed to be more readable

some shortcut actions, like zoom stuff, is no longer hardcoded!

mouse shortcuts are still not widely supported!

the shortcut command edit dialog now throws a veto-driven error message if you try to ok on an invalid command (a blank services choice or action string, that sort of thing)

ctrl+r is now 'remove_files_from_view' by default. the old 'show_hide_splitters' is now ctrl+shift+r by default

the preview canvas, if focused, now responds to many normal media viewer shortcuts (content stuff like archive/manage tags, and canvas-specific like frame back/forth, zooming and panning)

the new shortcut system now interprets double-clicks of any mouse button to be a second single click

the new shortcut system correctly 'flips' ratings on and off, rather than always 'setting' to the chosen value. optional 'set only' support will come in the near future

more shortcuts will correctly and reliably propagate to canvases when any part of a hover frame has focus

fixed a conflict between the new shortcut system and taglists, which were no longer accepting otherwise interesting keys, like enter

created a shortcuts manager cache that deals with a bunch of the shortcut workflow centrally

completely eliminated the old accelerator table/menu command system for the main gui window

lots of misc shortcut-related work

fixed wildcard file search predicates that are on a specific file service and have no namespace (like 'mar*')

certain routines that can cause mass refreshing of the menubar (like import folders) will now not spam (and often queue this spam up and hang the gui) the menu so much. the menubar will now always collapse multple overlapping refresh calls to reduce cpu load

fixed a focus-None issue in the new hover window focus detection code

improved some more focus detection and comparison logic--quick rating-scrolling in the media viewer should be less janked

improved 'touch' drag event detection and improved media canvas cursor hide/show logic in general

fixed '&' display in notebook page names

fixed '&' display in some common dialogs' text

wrote a new statictext class that deals with '&' better and in future will autowrap and maybe some other stuff and then switched most of the simple instances of wx.statictext over

updated some out-of-date server help r.e. admin service initialisation

converted a little of the help on custom filter->custom shortcuts. I'll do more in future

wrote a simple checkboxlist dialog

misc cleanup

version 251

started shortcut overhaul by updating shortcut storage and underlying objects

shortcuts will now support multiple modifiers (ctrl, alt, shift) in some places

the regular shortcut entry control now supports mouse events through a radiobox (although only the duplicate filter will capture and deal with mouse events on the new system!)

on db creation or update, a new 'duplicate_filter' shortcut set will be generated that includes mouse shortcuts and new duplicate_filter commands

the duplicate filter now obeys this shortcut set

duplicate filter shortcut edit cog menu now works, and the active shortcut set will update if edited

wrote simple duplicate pair status update db code

added dupe filtering optimisation--given A > B, anything else better than A will be set to be better than B, and anything worse than B will be set to be worse than A

added dupe filtering optimisation--after any duplicate status change affecting A and B, any 'same file' siblings of A and B will receive the same relationship

the 'process now' button on review services is now gated by a yes/no dialog that better explains what's about to happen

the repository buttons on review services will disable when they can't do anything

repository update processing will now cancel mid-job much faster

reintroduced the 'service-wide update' button to tag services in review services

reintroduced the 'clear trash' button to the trash file service in review services

reintroduced num_files rating service reporting in review services

fixed mouse scroll wheel events from the ratings hover window not being correctly processed by the main media canvas

added some 'touch' event detection to try to better deal with media dragging through a touchscreen

import folders will now 'action' their import paths as they go, rather than only at the end of their import run

the new taglists are now better at remembering their selection through a content change

opening a file or path from a non-windows client should now create a non-child process for open/xdg-open calls that block (so closing the client should not then close the child process movie player application or whatever)

added unit tests for the new shortcut object

added unit tests for the new application command object

fixed unit tests for the updated shortcuts object

misc cleanup

version 250

improved 'file storage locations' help page descriptive text, including adding statements for actual current storage percentages

improved boot missing file storage error handling

wrote a 'repair file storage' dialog panel to manually fix missing folders during the boot phase

added some bells and whistles to the repair file storage dialog. it now shouldn't be possible to boot the client with an invalid file store--let me know how this works for you

added 'apply all siblings to all services' to options->tags

whenever combined-service siblings are consulted, local siblings will now always have precedence (so, you can now definitely overwrite any ptr siblings you don't like at the local level). this stuff remains a mess, however. more work is needed

fixed searching for 'like' ratings

rating predicates will now render their value as 'like'/'dislike' or '3/5' rather than the underlying float

fixed a variety of predicate string rendering errors when 'show namespace' is false

fixed delete key not firing on autocomplete text entry controls

the autocomplete entry's text box will now take home/end/up/down events when there is only one result in the dropdown, when navigation of that list is not important (previously, this was only true when there were no results)

fixed an issue where invalid typed 'system:' tags were not being correctly cleaned

improved tag cleaning code

fixed fetching autocomplete results for small and specific searches (e.g. putting in 'a' will return results just for 'a'

fixed small and specific searches to return namespaced instances as well (e.g. putting in '8' can now return 'chapter:8'

fixed an issue with showing collected media in the preview window

fixed the file import status button not laying out correctly on edit subscription dialog

cleaned up some focus checking code

harmonised all initial key event processing

cleaned some key event processing

adjusted some server daemon timings to prefer post-boot update generation catchup over db maintenance

corrected a bunch of slightly mistaken unit test code

added some 'like' and 'numerical' ratings search unit tests

the advanced content update dialog now uses the new sizing system (so the go! button should no longer wander out of frame!)

reduced what I believe is excess fuzzy padding on the new sizing system

optimised a common client db request, speeding up a number of tag operations

gui now asks for confirmation when you make to delete a gui session

improved some out of date unit tests

version 249

reintroduced shape and colour options to edit ratings service panels

reintroduced num_stars and allow_zero options to edit numerical ratings service panels

the export phrase--defaulting to '{hash}'--will now persist through export dialogs. it is saved whenever you click update or export

removed the unintended 'counts' that were appearing after related tags

fixed 'remove' action in custom filters

hitting the delete key on manage tags taglist now will always remove

added a BUGFIX option to not verify regular https traffic on the old networking engine for those non-Windows users who are getting SSL verify errors

refactored the top hover frame more, making it more flexible

created a new top hover frame for the duplicates filter

duplicates filter now also supports tag and ratings hover frames

duplicates filter now reports A or B as file index

added some placeholder buttons to the duplicates filter for actions and 'cog' customisation

reduced duplicate search shutdown logspam

improved duplicate search job cleanup

improved mime detection of thumbnail regeneration

improved some shutdown error handling

improved numerical rating search accuracy

improved some hdd import error handling

improved some unicode error handling

improved reliability of some database transaction processing

fixed animationbar having problems with single-frame videos

fixed namespace colour taglist not updating colour correctly

stopped the namespace colour taglist from deleting default namespace colours

made the new class of button more compact--not sure if I like it

refactored some db initialisation to avoid future transaction problems during special updates

tidied up some last incomplete taglist code from last week

did a little prep for some future shortucts overhaul

did some more menu code updating

added a simple subscription save/load unit test

moved outdated server unit tests forward

fleshed out some server certificate generation

some other misc v245 catchup work

cleaned up some pydeadobjecterrors caused by downloaders reporting progress to destroyed windows

version 248

fixed two more issues with recent update code!

re-applied a more rigorously tested 'invalid subtag' replacement for any user that has experienced incomplete update code in the past few weeks

added some tag rendering options to options->tags that affect most user-facing, non-editing windows:

added custom namespace-subtag connector (i.e. instead of ':')

added 'show namespaces' checkbox (i.e. to keep the colour but lose the namespace, like how most boorus display tags)

created a basic framework for the duplicate filter--it doesn't do anything yet, but you can launch it from the duplicates page to check it out

updated 'reset processing cache' action and added it to review services repository pages

review services now always starts on 'my files'

review services will try to stay looking at the currently selected service through a refresh_services event

fixed some thread-gui refresh interaction in review services

all listctrl-attached delete/remove buttons will now ask if you are sure before they go

reenabled db->maintenance->clear orphans, which it turns out I had already fixed by accident

serverside services will lock themselves more efficiently while generating updates, ensuring 'server busy' responses can always return promptly

fixed tag manage parents a/c entry to remove tags when they already exist in the box (previously, the a/c would only ever add, so a double-click was required to remove)

tuned adminside petition processing so it approves/denies in reasonable-sized chunks

refactored all the taglist gui code to focus on the underlying tag rather than the string representation

plenty of misc taglist cleanup besides

started a unit test suite for listboxes and wrote a thorough test for namespace colour listbox

misc canvas refactoring

version 247

fixed a problem with deleting more than 256 files at once

furthermore, deleting from thumbnail view or after a filter will split delete jobs into chunks of 64 files at a time to reduce gui hang from deleting many tag-heavy files

rewrote canvas media container code to recycle containers, embed windows, animation bars, static image windows, and animation windows. scrolling through all kinds of media is less flickery (less 'grey box' window initialise flicker) and scrolling through static images should be completely flickerless! William Gibson slideshow speed works again!

this flickerless static image transition will be particularly useful in the forthcoming duplicate image filter!

got adminside petition processing working again

petition counts and fetching is now split by content_type and status

the approve/deny colour hint is more obvious on the petition panel

petitions now process off the main gui thread and throw up a popup message

added 'check all' and 'check none' buttons to petition panel

several serverside petition processing fixes

generalised and improved dynamic menu check item initialisation and inversion support

moved the 'get tags even if file already in db' option into a cog button on regular downloader pages

added a default menu option for 'get tags even if file already in db' to the same cog button

added this cog button to the edit subscription panel as well

fixed exporting tags to Hydrus Tag Archives

fixed exporting 'all known tags' to HTAs

cleaned some HTA and related code

fixed namespace based tag censorship

fixed autocomplete not filtering out current/pending counts if they are set to 'excluded'

fixed generation of non-expiring new accounts

fixed some v245->v246 tag improvement code that was replacing invalid tags with the incorrect namespace

patched a problem with '-:' dirty tag in the v245->v246 update code--I'm not sure what it was doing, but it catches the unusual problem and puts it in the 'invalid tags' category, so let me know if you get trouble with this in future

wrote unit tests for bytesdictionary and shortcuts serialisable objects

harmonised and improved how separators are appended to menus

cleaned up some client db index creation

cleaned up some client tuple stripping

misc pylint warnings cleanup

misc fixes

version 246

fixed a critical bug in serverside content deserialisating that meant servers were not processing most client-submitted data properly

fixed a critical bug in 'all known tags' autocomplete regeneration--please run database->regen->a/c cache when it is convenient and let me know if your numbers are still off

fixed the pre-v238 update problem for good by abandoning update attempts and advising users to try v238 first

clientside invalid tags will now be collapsed like with the server last week. if a tag is invalid (typically something with an extra space, like "series: blah"), the update code will attempt to replace existing mappings with the collapsed valid version. some unusual cases may remain--they will be replaced with 'invalid namespace "series "' and similar. Please remove and replace these when convenient and contact me if there are way too many for you to deal with

duplicates pages now have a file domain for the filtering section, and they remember this domain through session loads

this file domain is accurate--counting potential duplicates and fetching pairs for 'show some pairs' only from those domains. the issue of remote files appearing should be gone!

there is now only one 'idle' entry in the duplicates page cog menu--it combines the three previous into one

fixed numerous irregularities across the wildcard code. all search input now has an implicit '*' on the end unless you put a '*' anywhere else, in which case it acts exactly as you enter it, with a non-* beginning matching beginning of string, whitespace, or colon, and non-* end matching end of string or whitespace

autocomplete now searches namespace, so entering 'char' will load up all the 'character:' tags along with 'series:di gi charat'. this can lag to hell and back, so it may either need some work or be optional in the future. feedback would be appreciated

typing 'namespace:' will include all the series tags below the special optimised 'namespace:*anything*' tag

autocomplete searches recognise an explicit '*' no matter where it is in the entry text. typing 'a*' will load up all the a tags and present a 'a*' wildcard option

quickly entering a wildcard entry will now submit the correct wildcard predicate ('rather than a literal 'hel*' or whatever tag)

review services panel now reports total mappings info on tag services

review services panel now reports total files info on file services

manage services's listctrl is now type | name | deletable and initially sorts by type. the strings used for hydrus service types are also improved

manage serverside services (called by server admins to manage their services) have fixed setnondupe port and name on edit service events

new popup messages will now also appear if there were previously no popup messages to display if the current focus is on a child on_top frame, such as review services (you'll now see the processing popup appear when you click 'process now' on review services)

the popup message manager now initialises its display window with a single message that is quickly dismissed. this helps set up some variables in a safe environment so they don't have to be generated later when the gui might be minimised or otherwise unusual

hid hydrus update files from 'all local files' searches

added 'media_view' entries for hydrus update files, just in case they are still visible in some unusual contexts (and they may be again in a future update anyway)

fixed 'recent tags' being returned from the database out of order

by default, 'recent tags' is now on for new users

'get tags even if file already in db' now defaults to False

file import status now allows a 'delete' action below the 'skip' action

file import status right-click event processing is more sane

fixed the new raw width/height sort choices, which were accidentally swapped

cleaned the media sort code generally

cleared out some redundant rows that are in some users' client_files_locations

namespaced predicates are no longer count-merged with their namespaceless versions in 'write' autocomplete dropdowns

'unknown' accounts should now properly resync after clientside service change

improved how registration keys are checked serverside when fetching access keys

fixed a v244 update problem when unexected additional tag parent/sibling petitions rows exist

improved my free space test code and applied it to the old v243->v244 free space test (it'll now test free space on your temporary path and report problems appropriately)

to improve log privacy and cleanliness, and to make it easier to report profiles, db/pubsub profiles now write to a separate log file named appropriately and labelled with the process's start time

profiles are more concise and formatted a little neater

across the program, three-period ... ellipses are now replaced with their single character unicode � counterpart (except to the console, where any instance of the unicode ellipsis will now be converted back to ...)

cleaned up some log printing code

cleaned up some experimental static serialisation code, still thinking if I like it or not

started on some proper unit tests for hydrus serialisable objects

fixed and otherwise updated a heap of unit test code to account for the v245 changes

cleaned up a bunch of old database table join code

started some databse-query-tuple-stripping code cleaning

deleted more old unused code

misc timing improvements

misc code cleanup experimentation

misc cleanup

version 245

fixed a v244 update problem for clients updating from <v238

some misc stuff:

if you start editing many subscriptions, cancelling a single dialog will break the chain of loading new dialogs

reduced some redundancy in regular client file import

improved how the dialog for selection from a list of strings works

if the client ever merges one directory to another (such as in external file locations migration) and any files fail to merge, the source will no longer be deleted

created new flexible bandwidth tracking and ruling objects

updated how repository updates work, splitting the old explicit and self-contained content update package system to a new implicit definitions/content split system and an improved one-step-sync metadata propagation

updating now takes approximately 23% as much bandwidth as before

update files are now stored in client_files and server_files like any other file (client_updates and server_updates folders will be deleted on update)

the server will print update generation info to its log

unfortunately, updates cannot be converted, so a complete resync of update files is required. the smaller update size and better bandwidth controls should mitigate the problem somewhat

the server has been compacted across all content types--its mappings db file should shrink about 12%

due to new service-specific ids, server.master.db should increase in size, typically about 50%

harmonised how GET and POST/response args are built and parsed across the network

server administration initialisation is now simpler, done with 'init' registration key

leading and trailing spaces are now removed from both the namespace and subtag components of namespaced tags, meaning 'title: blah' will be collapsed to 'title:blah'. tag repositories will clean their existing tags on update

invalid serverside tags will be replaced with valid placeholders on server update

refactored, harmonised and simplified some server request code

Eris makes a triumphant return to the root welcome page ('/' request of any service) with improved self-description text as well

updated how bandwidth is tracked and overseen

all requests now consume bandwidth

fewer requests are actually constrained by bandwidth--at the moment, it is only update files and file/thumbnail downloads, as these represent the overwhelming majority of bandwidth consumption and are not at all critical to service operation--this may change in the future, but it suits our purposes for now

bandwidth tracking code is more sane across the board

the server's administration service now tracks server-wide bandwidth as it happens

service and server-wide bandwidth rules are consulted as soon as the request begins

improved some server response rendering

permissions are more flexible and content-specific

improved how file repositories check and process file requests

created a gui control for managing bandwidth rules

updated the serverside service object

the serverside service object now contains bandwidth rules and tracking

updated how the server deals with services on a db level

refactored and cleaned a ton of server db code

updated how services are edited over the network

updated how the server associates its services with its http pipeline

converted clientside server service management gui to the new panel system

updated gui code for clientside server service management

converted clientside server service management db code to the new system

improved how bandwidth errors are reported

updated the account type object

the account type object now contains bandwidth rules and a more flexible permissions system

updated how the server deals with account types on a db level

converted clientside server account type management gui to the new panel system

updated gui code for clientside server account type management

serverside account types are now fetched from a cache, reducing memory sprawl

updated the account object

the account object now contains a bandwidth tracker

updated how the server deals with accounts on a db level

updated the clientside server object

the clientside server object now contains bandwidth rules and a tracker and has improved error management, recovery, and reporting

updated how the client deals with services on a db level

improved how serverside bandwidth errors are caught

all clientside services start with a 50MB/day bandwidth limit

existing repositories will get a 250MB/month, 50MB/day default bandwidth limit, just to help us get over the hump--see the release post for info on how to get all the updates anyway

updated manage services dialog extensively

service account registration now occurs through a simple button on the normal clientside service edit panel

all services can now be renamed

updated the content processing pipeline

prepared code for future merging of file and tag repositories

added future support for pend-petitions for files and mappings and simple creation permissions for tag siblings and parents

the clientside content processing pipeline now operates inside a single database transaction, reducing a great deal of previously redundant hard drive activity

the serverside content processing pipeline now creates per-service timestamped content definitions

the clientside content processing pipeline now maintains and appropriately consults a cache for server definitions

pre-process disk cache is more intelligent

updated review services to match the new service objects

service review panels now show more information about error state and so on

service review panels should now update as soon as services change

you can now force sync repositories from review services

you can now pause/resume repositories from review services

you can now export repository update files from review services

you can now import repository update files from the services menu

repository thumbnail download now uses a cache for faster thumbnail ownership testing

culled a lot of old code and experiments

deleted all the old messaging depot code and db table cruft

removed optimised petition processing

serverside deleteorphans is temporarily disabled

hydrus client-repository relationship no longer supports news

removed the 'stats' admin service call--it will come back as a account review page

clientside clear orphans is temporarily disabled

clientside local file/thumbnail server is disabled for now

custom service 'messages' for the root page are no longer available

some things are not working yet--they will be back in soon

content presentation on review services

service reset, other advanced content service controls

ipfs controls on review services

local booru controls on review services

most service-specific panels on manage services

petition resolution

serverside account modification, including banning

version 244

updated client database to compact ( namespace_id, tag_id ) pair into a single id for storage

added some bells and whistles to the update code

added a free space check and messagebox warning before the update

updated db, service, and a/c cache creation code to reflect new schema

updated absolutely everything else in the db to reflect the new schema

for users with plenty of tags, the db should now be about 33% smaller!

unified how unnamespaced tag searching counts are totalled

unnamespaced tag searching counts are now totalled when the tags are fetched from the in-view ui media

unified how tags are split into ( namespace, subtag ) across the program

fixed deviantart gallery thumbnail parser

fixed linux session load page key event handling bug

os x can now support notebooks with zero pages open

fixed an issue where os x was losing the first page of some session loads

fixed some similar files shutdown work false positive calculation

reduced server bandwidth check period from 24 hours to 1 hour

improved calltothread scheduling under heavy load

improved scheduling of how files are physically deleted

numerous laggy temp_table replacement/cleanup

more temp_table replacement

misc efficiency improvements and general db code cleanup

misc path code cleanup

version 243

updated more menu code to the new system

finished updating all main gui menubar and thumbnail menu code

all brackets, braces, parentheses, and single and double quotes are now ignored when matching search text to tag. inputting 'mercy' matches 'character:angela "mercy" ziegler' and entering '[[[intensifies]]]' matches '[intensifies]'

'*' searches will now return no autocomplete results, rather than trying to load absolutely everything

made wildcard searching significantly more permissable--it now implicitly applies a '*' to the beginning as well as the end of your search input

the main gui's notebook pages will now be disabled while a session loads

might have fixed a first-boot layout issue--otherwise added a layout entry to the debug menu for further testing of this problem

improved progress layout on file import dialog

created a more efficient way for threads to update gui elements

the file import dialog uses this new update system and parses lots of small files much faster as a result

the edit import folder dialog will now complain if you try to put in an import path that includes the install or db directory

ffmpeg mime parsing now catches and reports audio formats without any hassle

thread downloader no longer fetches file extension in filename tag 'filename:blah' instead of 'filename:blah.jpg'

changed png default zoom rules to copy jpg--the unusual zoom as default was more confusing than helpful

updated sqlite for windows--should be a decent bit faster

deleted a lot of old code that'll never be needed again

misc refactoring

misc cleanup

version 242

optimised 'exact match' similar file queries to run a lot faster

optimised similar file queries in general, particularly for larger cycle queries

optimised hamming distance calculation, decreasing time by roughly 45%!

the similar files tree maintenance idle job will not trigger while there are phashes still to regenerate (this was redundantly and annoyingly blatting the new dupes page as soon as phash regen was paused)

removed similar files tree maintenance entry from db->maintain menu, as it can be done better from the new dupes page

adjusted the duplicate search and file phash regen progress gauges to reflect total number of files in cache, not the current batch job

all maintenance jobs on the duplicates search page will now save their progress (and free up a hanging gui) every 30 seconds

the duplicates page's cog menu button now lets you put phash regen and tree rebalancing on the normal idle routine, defaulting both to off

the cog menu can also put duplicate searching on idle time!

added a very rough 'just show me some pairs!' button to the dupe page--it is pretty neat to finally see what is going on

I may have reduced the memory use explosion some users are getting during file phash regen maintenance

wrote an unclose_page action and added it to the shortcuts options panel--it undoes the last page close, if one exists. ctrl+u will be the default for new users, but existing users have to add it under options

added ascending/descending sort choices for width, height, ratio, and num_pixels

the client can no longer talk to old http hydrus network servers--everything is now https

in prep for a later network version update, the client now supports gzipped network strings (which compress json a lot better than the old lz4 compression)

fixed gif rendering in the Windows build--I forgot to update a build script dll patch for the new version of opencv

the export file dialog's neighbouring .txt taglist file stuff now allows you to select a specific combination of tag services

if an hdd import's original file is due to be deleted, any existing neighbouring taglist .txt file will now also be deleted

the inter-thread messaging system has a new simple way of reporting download progress on an url

the handful of things that create a downloading popup (like the youtube downloader) now use this new download reporting system

sankaku seems to be 503-broke due to cloudflare protection--I have paused all existing sankaku subscriptions and removed the sankaku entry for new users (pending a future fix on my or their end)

I've also removed danbooru for new users for now--someone can fix the long-running sample size file issue in the new downloader engine

removed unnamespaced tag support from the hentai-foundry parser--maybe someone can try to fix that mess in the new downloader engine

menubuttons can now handle boolean check menu items that are tied straight into hydrus's options

menus launched from the newer frame and dialog code will now correctly display their help text on the main gui frame's statusbar! (at least on Windows! Linux and OS X remain borked!)

fixed a unicode error parsing bug in the gallery downloader

the server stop (or restart) command now correctly uses https!

the server test code now works on https as appropriate

fixed some misc server test code

misc fixes

misc cleanup

misc layout cleanup

version 241

fixed the 'setnondupename' problem that was affecting 'add' actions on manage subscriptions, scripts, and import/export folders

added some more tests to catch this problem automatically in future

cleaned up some similar files phash regeneration logic

cleaned up similar files maintenance code to deal with the new duplicates page

wrote a similar files duplicate pair search maintenance routine

activated file phash regen button on the new duplicates page

activated branch rebalancing button on the new duplicates page

activated duplicate search button on the new duplicates page

search distance on the new duplicates page is now remembered between sessions

improved the phash algorithm to use median instead of mean--it now gives fewer apparent false positives and negatives, but I think it may also be stricter in general

the duplicate system now discards phashes for blank, flat colour images (this will be more useful when I reintroduce dupe checking for animations, which often start with a black frame)

misc phash code cleanup

all local jpegs and pngs will be scheduled for phash regeneration on update as their current phashes are legacies of several older versions of the algorithm

debuted a cog menu button on the new duplicates page to refresh the page and reset found potential duplicate pairs--this cog should be making appearances elsewhere to add settings and reduce excess buttons

improved some search logic that was refreshing too much info on an 'include current/pending tags' button press

fixed pixiv login--for now!

system:dimensions now catches an enter key event and passes it to the correct ok button, rather than always num_pixels

fixed some bad http->https conversion when uploading files to file repo

folder deletion will try to deal better with read-only nested files

tag parent uploads will now go one at a time (rather than up to 100 as before) to reduce commit lag

updated to python 2.7.13 for windows

updated to OpenCV 3.2 for windows--this new version does not crash with the same files that 3.1 does, so I recommend windows users turn off 'load images with pil' under options->media if they have it set

I think I improved some unicode error handling

added LICENSE_PATH and harmonised various instances of default db dir creation to DEFAULT_DB_DIR, both in HydrusConstants

misc code cleanup and bitmap button cleanup

version 240

improved how the client analyzes itself, reducing maintenance latency and also overall cpu usage. syncing a big repo will no longer introduce lingering large lag, and huge analyze jobs will run significantly less frequently

the analyze cache will be reset on update, so you will have one big round of analyze the next time you maintain, and then you are good

added data structures to support auto-discovery of duplicate files

improved how some similar files maintenance occurs

flushed out duplicate status reporting

added a new page type, currently under pages->search pages->duplicates, to handle duplicate discovery and filtering

created a gui skeleton for the new duplicates page

started some handles and update code for the new duplicates page

wrote a new txt file in the db dir about the new emergency extract scripts

wrote an emergency extract script to migrate subscriptions to a new db

wrote an emergency extract script to migrate options to a new db

the trash clearing daemon now runs in the foreground, and a little of its code is improved

the trash clearing daemon now makes a popup message when it does work

the server's ssl keys are now set to read-only on Windows and user read-only only (i.e. chmod 400) on Linux and OS X on creation and update

added a explicitly unicode popup message to the debug test

fixed some network error catching code that was using Windows-only error codes

converted more of the thumbnail right-click menu over to the new system

improved some listctrl code

misc cleanup

version 239

finished up similar files search data maintenance code

similar files search data maintenance will now run during idle time

similar files search data maintenance can be called from database->maintain menu

the crowded database->maintenance menu is now split into maintain, regenerate, check

improved the similar files tree generation code, speeding searches significantly

wrote a new listctrl class to handle more complicated objects and also sort by underlying data

the new listctrl now handles object name non-duplication

cleaned a bunch of crap old listctrl code

manage export folders now uses the new listctrl

manage import folders now uses the new listctrl

manage subs now uses the new listctrl

manage scripts now uses the new listctrl

options media viewer options now uses the new listctrl

file import status panel now uses the new listctrl

the new listctrl can now quickly fetch item index from the underlying object

the file import status panel should now cope with extremely huge lists a bit better now!

multiple parsing child nodes can now import from/export to clipboard as lists

export folders now have names, so you can have multiple export folders pointing to the same path! existing export folders will get their path as their name, but this can be changed no prob

cleaned import/export folder dialogs

cleaned import/export folder dialog workflow

several misc import/export folder improvements

hydrus servers are now exclusively https with self-signed certificates

hydrus servers now create server.crt and server.key in their db folders for SSL--these files will be backed up along with everything else on an admin backup command

system:hash now ignores the file domain and any other predicate. it now returns very quickly, no matter the context

improved system:hash search logic

all the awkward choice dropdowns in system predicate panels are replaced with radioboxes

improved system:rating panel grid layout

wrote a better subclass of radiobox to handle more data

moved first half of thumbnail menu to new menu system

cleaned up a little thumbnail menu logic

improved the different ways services are added to thumbnail menu

thumbnail 'select' menu is logically cleaned up and allows for better file domain selection

the thumbnail menu's copy files and copy hashes to clipboard will now send them ordered (they were previously pseudo-random)

added 'paths' to the share->copy thumbnail menu for copying multiple files' paths. these are also ordered.

if the popup message manager does not have any errors, it will no longer unhide (which can annoyingly raise the main gui window) when the gui window does not have focus

removed some old redundant error reporting stuff in popups

improved and quietened some some mime detection failure code, sped up mime failure loop in all cases

massively simplified and atomised how new serialisable-object management panels can save their data

the manage subscription dialog now saves in a single, faster transaction

the manage script dialog now saves in a single, faster transaction

reduced redundant index work from analyze jobs

improved tag parse error handling

fixed media removal rules when deleting from the 'all local files' domain

polished and clarified some of the help's tag schema

misc cleanup

more misc cleanup

misc refactoring

more misc refactoring

version 238

added left/right equal ratio preference to the similar files search tree generation

added high standard deviation preference to the similar files search tree generation

added 'db profile mode' to the help menu, for briefer profiling in future

when a similar files search occurs with db report mode on, it'll popup a nice bit of text saying how many cycles it took

fixed a bug due to last week's 'all local files' change that meant archive/delete filtering would include any selected trashed files

'recent tags' will now only be populated by add/pend tag events, and only by those tags that were valid for that action

moved the help menu to the new menu system

added bitmapped menu item support to the new menu code

fixed and cleaned up some of the new menu code

improved a/c cache regen error reporting

version 237

added 'all local files' service that spans all local file domains

improved trash service code

trashed files now report their trashed timestamp in right-click menus

trash views will sort oldest/newest by the trash timestamp

renamed 'local files' to 'my files' to reduce initial confusion

if you don't like 'my files', you can now rename the local files service under manage services!

improved how some local file service metadata is stored

cleaned and possibly fixed up some delete code

cleaned up a bunch of misc service and file service code

the client is more intelligent about what local files are -- you can now 'open externally' trashed files, for instance

on multiple monitor systems, the new sizing system now bounds itself by the appropriate monitor's dimensions (previously, it was always consulting the primary, I think)

if an expansion event causes a frame to grow off screen, the new sizing system will now attempt to move it up and left so it is completely visible

fixed an important bug in the specific service autocomplete cache that was leading to several kinds of miscount--please regen your autocomplete cache at your convenience

the client can now post additional messages on boot (you'll see one!)

improved how errors with unusual characters are applied to failed import file status objects

left double-click on the main gui greyspace now opens the new page chooser as well

restored session pages now recover more gracefully from missing services

shortened the 'stop after this many files' phrase, which was mis-sizing downloader panels

changed several bits of old jank in the 'import options - files' collapsible. it is also now thinner

misc gui improvements

updated more of the menu system code

the selection tags box now sizes its height more reasonably

fixed 'check now' and 'reset cache' buttons on edit sub dialog

subscriptions report better file import status on their popup

misc cleanup

version 236

in prep for a network https upgrade, the client can now detect and escalate to https when making connections to hydrus services

import/export to png and clipboard now supports multiple objects at once!

rewrote the manage subscriptions dialog to work on the new panel system

the new manage subscriptions dialog has a listctrl and a sub edit dialog

the new manage subscriptions dialog has the same add/export/import/dupe/edit/delete buttons as the manage scripts dialog

subscriptions are now importable/exportable, including en masse with the new multiple object import/export support!

the new manage subscriptions dialog has retry failed/pause-resume/check now/reset buttons for easy mass subs management

the edit subscription panel has a bit of a layout makeover

the edit subscription panel now updates itself as its buttons are hit

the edit subscription panel disables buttons that are not applicable

subscriptions can now be renamed!

cleaned some misc subscription code

relabelled initial and periodic file limit in the subscription edit panel

middle-clicking on the main gui's greyspace (e.g. to the right of the notebook tabs) will spawn the new page chooser!

created a simple HydrusRatingArchive class--will do more with it in future

added ffmpeg, python, and sqlite versions to the help->about window

harmonised daemon code

added a new class of daemon that will not fire while a session load is occuring

subscriptions, import and export folders, and file repo downloads now use this new daemon

cleaned the way background daemons check for idle

expand/collapse panels now notify the new kind of toplevelwindow that a resize may be needed when they switch state

time deltas (like on subs edit panel or a thread watcher) now render more concisely ('7 days' instead of '7 days 0 hours')

serialisable object png export panel now has a width parameter

fixed a bug where tags that begin with unicode digits were accidentally identifying as numbers for the purposes of sorting and throwing errors on convert fail

the media viewer can handle some more unusual content update combinations--for instance, if it cannot figure out which media to show next, it will revert back to the first image rather than displaying an undefined null mess

updated and cleaned a bunch of my old misc encryption code

misc cleanup

version 235

finished first version of new faster dupe search--'system:similar to' now uses it

finished faster dupe search tree creation

finished faster dupe search tree search

added faster dupe search tree leaf insertion

added faster dupe search tree rebalancing maintenance

added faster dupe search tree orphan deletion maintenance

added faster dupe search leaf-regeneration-as-scheduled maintenance

misc faster dupe search tree work

new search tree will be created on db update

fixed the 'add' buttons on the import files dialog

fixed linux manage tags text input

reduced incidence of linux media viewer post-manage tags key event swallowing

fixed an issue where if the manage tags dialog was launched from the media viewer while a hover window was displaying, the main gui would often be raised when the manage tags frame was closed, annoyingly obscuring and de-focusing the underlying media viewer

fixed manage tags action workflow when no valid tag choices exist

entering a tag petition reason now has several common 'suggestions' for quick entry

removed ffmpeg binary from default linux release (as basically every linux has a decent ffmpeg by default)

fixed the incorrect ffmpeg version for os x

added an option to options->media to prefer to use the system ffmpeg over any in the hydrus bin directory

fixed linux arrow key navigation of the page picker dialog

cleaned up the page picker dialog's gross old code

the edit file lookup script panel now has a script management control on its test page to better report on and control tests

scripts will new correctly attempt to decode unicode and other encoded text in responses

if non-text data is accidentally fetched in the file lookup script panel's test page, it is now caught and a small summary of that event is printed to the text box instead

added the same non-text catch to the edit link node panel

improved how some script errors and status updates are handled so the new test window script management controls will see and report more useful information

removed 'restart' option from linux for now--looks like a bug in pyinstaller

improved how the new GET request read reattempt works

cleaned up some spammy error code

cleaned up spammy test code

misc error handling improvements

misc cleanup

version 234

the action choice workflow in the manage tags dialog now merges decisions for multiple tag entry events

the button dialog that pops up on the new merge manage tags workflow has button tooltips to better describe the proposed action's tag and file combinations

the manage tags dialog only requests one petition reason for an above merged multitag petition event

perceptual hashes are now stored in the cache db (moved from the preferably leaner main db)

the database now supports multiple perceptual hashes per file

fleshed out the perceptual hash vptree generation and maintenance code

added option for main gui title to options->gui--it even updates live

added unicode path support when importing serialised pngs

added unicode path support when exporting serialised pngs

exporting a serialised png now reports success via the export button, which will temporarily relabel itself

added discord links to help files and help->links menu

when a GET network connection fails during the read phase due to an unexpected timeout, the request will be reattempted a couple of times, like failed connection initialisations currently are

escape key now closes scrolledpanelframes (review services or import status frames)

manage tags dialog and frame now closes due to escape key correctly

fixed a size calculation bug that was not initially drawing scrollbars on manage options and any other listbook-containing scrolling panel when the screen is too small to show the whole dialog

undo menu now works on the new menu system

cleaned up some bad gui-thread interaction in the file import dialog

fixed file->restart in the built release, including when the install path includes a space

the trash service no longer records which files it has physically deleted, as this information is not used and is redundant compared to the existing local files' deleted record (existing records will be deleted on update)

the subscription daemon will wait 90 seconds after boot before triggering--quitting the client before then will result in subs not being checked

version 233

made a plan for faster dupe search

created skeleton of db tables for faster dupe search

wrote out search algorithm for faster dupe search when the rest is ready

updated search algorithm and skeleton to support a future multiframe (i.e. gif/video) similar files comparison

the os x release now has 'client' as the main client executable

the os x release now includes the server, under the 'server' executable

added studio (128,0,0) and meta (0,0,0) default namespace colours

the 'add all' button on the file lookup tag suggestions panel now will only ever add--it won't remove/rescind pend

the password system now supports non-ascii input (be careful though, as even a subtle change in keyboard encoding that nonetheless may have the same visual characters will likely be considered a different password)

fixed the 'nonetype has no dtype' rendering problem introduced by last week's 16-bit channel fix (this mostly affected static gifs)

fixed non-null ratings changes in the numerical ratings dialog

fixed culling and adding variables not initialising on the edit html formula panel

fixed the htmlparser attribute fetcher to deal with both single value attrs ('id') and multiple value attrs ('class'). in the latter case, the many values will be joined up, as how they appear in html

tweaked disk cache timings a little more to account for more scenarios--it now also reports itself to the shutdown splash screen

version 232

finished png object sharing system, created some gui for it

switched the top half of the new png export to wx code, which can render with prettier system fonts

you can now 'export' parsing scripts to png from manage parsing scripts for easy sharing

you can also 'import' parsing script pngs in the same dialog!

the file lookup tag suggestions panel now listens for media update events (so if you browse the media viewer with manage tags open, it'll keep up with the current file)

added a 'add all' button to quickly add all tags from file lookup suggested tags panel

created a 'script management' control for file lookup suggested tags panel and added some script text status updates for it (progress gauge doesn't do anything yet)

added basic 'cancel' support to script management control--mid-http cancel support will come with the gauges

added url support to thread notification objects, and hooked script code into it

added a url menu button to script management control, so you can launch any parsed urls the script found in your browser

html parsing formulas now support positive and negative front and end character culling

html parsing formulas now support prepending and appending arbitrary text

added some more pretty text to the html formula summary to explain when they do the new culling or adding

script 404s and other network errors are now caught neatly and reported

link node 404s and other network errors are now reported and recovered from

fixed favourite file lookup script selection on initialisation

fixed parsing veto test to include membership of search string in result, not just exact match

fixed html formula tag attribute fetching

updated os x release to python 2.7.10

fixed os x about window

fixed v231 linux release image rendering crash

updated linux release to wx 3.0.2.0

updated linux release to opencv 3.1.0

fixed rendering for 16-bit-per-channel images

fixed a potential out-of-order settagboxfocus event issue on panel initialisation

created a 'menubutton' control and switched existing code over

created some 'nullipotent panel' dialog code to handle some new stuff

misc fixes

bunch of misc refactoring

version 231

added file lookup scripts suggested tags control and appropriate options panel

added final bit of gui->filelookup script->content update pipeline conversion and tie-in code

moved all the suggested tag options into a notebook

added notebook layout for suggested tag control columns. it is now the default

suggested tag columns now have a unified set width

the parsing ui's add script and node buttons now spawn menus rather than the awkward listofstrings dialog

added 'iqdb danbooru' file lookup script. it hangs the ui pretty bad atm, but that will change in future

fixed a critical pyinstaller problem with the os x builds

log files will now be appending with 'year-month', and will roll over to a new file as the month turns

help->about dialog now has some library version information

converted more menu stuff to the new system

wrote the guts of a new png-based object sharing system

dropdown choices are more resistant to missing init and invalid defaults

tweaked some disk cache maintenance timing

fixed a multi-version update bug regarding external thumbnail paths not being initalised

misc cleanup

misc cleanup, improvements

version 230

all multiline text ctrls now support ctrl+a for select all, wew

added dynamic menu_item tracking to new menu id system

added submenu tracking to new menu id system

added comprehensive menu, submenu, and menu item destruction to new menu id system.

'pending' menu now works on the new menu id system--is likely a major cause of the menu id bugs

added 'info' tabs to the script/nodes/formula parsing gui panels

added a 'veto' content type for discovering undesired redirects

added special veto support to parsing engine

expanded the content node panel to handle and display multiple content types

added 'link' parsing node panel

expanded 'link' panel test page and tied in secondary layer example url and data forwarding to children

fixed the parsing node children control not forwarding example data up to nodes on edit

file lookup url is now explicitly stored in the file lookup script object

file lookup url is now passed to child parsing dialogs for test example data purposes

expanded internal 'requests' interface a little--it now also always includes the 'user-agent' request header as the typical 'hydrus/version_number'

wrote script GET/POST engine gubbins and tied it into the respective test page

wrote link GET engine gubbins and tied it into the respective test page

'person' namespace added to default namespace colours--it is RGB ( 0, 128, 0 ), a darker version of 'character' green

version 229

edit html formula dialog can now be launched from edit content node panel

edit html tag rule dialog can now be launched from edit formula panel

edit formula panel test now works

edit content node panel test now works

edit script panel test now works

html formula now filters out missing content rather than complaining

misc parsing work

fixed some listctrl resize columns to be more helpful

attempting to size a resizing listctrl column will now move the resizing column to the last column, which will avoid the flickering scrollbar sperg-out of previous

attempts to resize the final column of a listctrl while it is the resizing column will be silently vetoed, which will avoid more resize event flickering

frame location options panel now allows for negative position coordinates (for multi-monitor setups)

all the new resizing panels now handle certain size updates in a more unified and reliable way

fixed an issue with hover windows' mouse position check breaking during media window init/shutdown edge cases

avc1 mp4 support added

ftypFACE mp4 support added

ftypdash mp4 support added

mpegts mpeg support added

to help migration, paths are now only stored as portable (i.e. relative to the base path) if they are beneath the base path (previously, this was true for any path on the same partition, permitting PITA '../../my_files' portable paths)

to help migration, portable paths are now relative to db_dir, not installation base_dir (existing client_files_locations, ideal client_files/thumbs_locations, export_path, and HTA paths will be updated)

started work on cleaning up button code across the program

created a better way to bind menu events that shouldn't suffer the id overflow bug

started work on replacing the old id_to_action_cache menu mess with the new system

too-many redirect exceptions are now richer and caught more sensibly

circular redirects are caught as soon as they occur

reduced sqlite cache size (with the new disk cache, this is now less important)

client.py is now included in the source and windows frozen releases

version 228

added support for RIFF .avi files

fleshed out parsing ui, made progress on node editing, moved edit/test panels to notebook pages to reduce clutter

the video renderer now initialises off the gui thread, which should reduce some video browsing chunkiness

trying to save a session that would overwrite another (from the same name) now throws up a yes/no warning

reintroduced '--no_daemons' and '--no_wal' to the new command line argument parser

you can now set the default tag service for new search pages under the file->options->tags page

since it can now be anywhere, added 'db directory' to the file->open menu

fixed a build capitalisation issue that was making the windows frozen exe release crash when trying to run from source

the 'too many redirects' error message is expanded to print the url called and its redirect request

errors from trying to delete files in use by another process will no longer make popups--they'll just write a note in the log

improved how some errors are written to the log

misc fixes

version 227

both the client and server now use standard command line input and will produce a proper help with a -h switch (although the 'windowed' frozen client executable can't print back to console)

both the client and server now support a -d or --db_dir to set a db directory outside of the install directory. you can now run multiple clients off the same install!

critical boot error reporting is more reliable

if the admin port is in use on server boot, the error is handled better and the program will quit more easily

fixed mime parsing for some unusual webms

added support for even more unusual webms

added support for SonyPSP mp4s

fixed system:hash when searching for a non-existent non-sha256 hash

fixed layout of the 'dismiss all' popup button

preview windows will no longer update in the background if the 'hide preview window' option is set

misc fixes

misc cleanup

version 226

fleshed out a bunch of parsing script engine and gui framework and added an 'under construction' entry to the services menu. feel free to play around with it!

fixed autocomplete same-service sibling count merging

sped up how autocomplete counts are put together

results will build faster due to an improvement in how tag rows are fetched

results will build faster due to an improvement in how tag strings are fetched

incidence taglist sorting now secondary sorts by lexicographic in the 'correct' direction (a-z for desc, z-a for asc)

all listctrls now support ctrl+a to select all

cleaned up some superfluous and possibly buggy canvas zoom calc when using the open externally button

the zoom_switch event will now reset the drag coordinates (recentering the image) if the resultant zoom fits into the canvas frame

the mouse cursor hides itself more quickly on media drag on Windows (so warppointer jitter is hidden)

the popup message manager will be careful about switching from hide to show while the parent gui is minimised on windows, which should reduce the grey box problem

the popup message manager will specifically re-layout on a transition from hidden to shown to stop the 'crushed into a corner' layout problem that would sometimes happen here

popup message manager will now consider a minimsed gui as unfocused for the purposes the of the focus BUGFIX option

added a new delayed popup message item to the debug menu for testing minimised popup creation

vacuum maintenance period option is moved, reset to larger default of 30 days

'last session' will now always be updated on client close, even if it is not the default session

version 225

system:numtags is now much faster when applied to 'local files' or a file repository

system:numtags now correctly counts same-tag-different-namespace tags as distinct (for instance, [page:1, chapter:1] was previously being counted as only one)

refactored fast db integer iterable access code into a context manager

media result building is faster

added namespace-grouped incidental tag sorting

fixed the OpenCV image loader for monochrome images

fixed a thread interaction issue when drawing popup messages

popup messages should no longer flicker while static

popup debug test is richer

fixed editing of some taglists such as in explicit tags and import tag editing

SSL EOF errors are now caught by the networking engine

version 224

rewrote the static image rendering and caching pipeline -- images are now resized on the fly, and only the master image is cached

all image rendering and zooming is faster and consumes less memory

image rendering now obeys the zoom quality options in the 'media' options page!

static image zooming will take advantage of OpenCL (video card acceleration) wherever available

added an option to the 'media' panel to allow fast but potentially unstable opencv image loading

fixed the tumblr gallery downloader (tumblr put an extra character in their API response, wew)

the debug code profiler now prints more information

cleaned and updated reducing lag help page to match new profiler

'recent' suggested tags are now sorted by recency

the thread watcher now checks url history as well as md5 match to compensate for cloudflare optimisation making for unreliable api and causing dupe downloads

misc fixes

version 223

fixed the popup message manager's bad minimise recovery

added an option to automatically 'hide' the popup message manager on main gui minimise (which has patchy multiplat support--some mindow managers don't do this automatically, some do but break when you force it)

added an option to automatically hide the popup message manager on main gui dofocus, which may help with window managers that minimise to the system tray, which wx cannot detect

added a 'make some popups' test job to the help->debug menu

siblings will now present in a more service-specific way

many gui elements that display tags are now aware of their tag service for the purpose of collapsing siblings

simplified a whole load of the siblings code

in some conditions, media will load faster

autocomplete tag censorship will now also apply on in a service-specific way in 'all known tags' queries

another remote connection reset exception is now properly caught by the network engine

the embed button will now draw the correct background colour behind transparent thumbnails

the animation scanbar will no longer scan if it has not previously experienced a mouse_down event--it could sometimes inherit this status from a previous media filter or embed reveal, resulting in undesired instant scan and undefined animation canvas behaviour

advanced content update panel will no longer list 'copy' as an action if 'local tags' is the only tag service

fixed a bug when performing a 'go!' action in the advanced content update panel when 'local tags' is the only tag service

version 222

created a 'raw url' downloader page that just downloads urls and tries to import the result. it has a 'paste urls' button to make mass import of a list of urls easy

fixed an options update bug when updating to v221 from any version before v220

added support for 'ftypqt' quicktime (usually .mov) video

embed button now uses system gui colours

embed button now puts the thumbnail of the media, if one exists, behind the 'play' button

sped up an inefficient existing mapping check that was slowing new pending mappings for popular tags

'last session' will no longer be listed on the gui session delete menu

cleaned up the main gui's initialisation events--a sizing bug often triggered after system reboot may be fixed

popup messages are initialised in a safer way

popup messages are dismissed in a safer way

popup messages will hide/restore themselves more reliably when the main gui window is minimised/restored

the pending popup message queue is now regularly purged of already deleted messages

new popup messages will no longer raise the main gui window to the top

subscription http errors during the gallery sync phase are now caught and handled gracefully, with exact error text written quietly to the log

network timeouts during successful response read are caught and converted to a hydrus network exception that will be caught and handled more reliably up the chain

the client's upnp daemon will now silence upnp mapping errors that are due to the router being too busy or full or any other unknown errors. a simple statement about the error and an instruction to explore the problem with the manual upnp manager will be written to the log

finished flexgrid refactoring

the new automatic flexgrid creation detects subsizers and lines them up more accurately with standard controls

wrapped the different sections of the 'speed and memory' options panel into staticboxes

wrapped the misc crap up top the 'tags' options panel into a neater staticbox

taglists with unusual tags will copy them more reliably and present fewer invalid menu options

misc layout fixes

version 221

fixed the 8chan thread parser for the new sha256 file urls. legacy links should still work!

created a 'x recent tags' suggested tags control and all the db and so on to go with it

recent tags defaults to off--turn it on under options->tags

replacing the previously basic zoom_in_to_canvas, the zoom options now support 100%, max regular zoom, or canvas fit for scaling up and down and for the media and preview windows!

cleaned up some garbage zoom code

refactored and harmonised how default zooms are calculated

harmonised and improved how canvas zooms are set

zoom_switch now works for flash

if it looks like the last instance of the client did not shut down cleanly, a new client will now present a dialog to choose whether to attempt opening the default session or just a blank page.

all listctrls across the program have better secondary sort

added an isatty() stub to the HydrusLogger for any unusual error handlers wanting to call it (youtube parser pafy was doing this)

removed some accidental (123) counts from some predicate menu and clipboard rendering

refactored a bunch of redundant flexgrid creation code

did a bunch more flexgrid refactoring

refactored some ugly global variables

misc mvc cleanup

misc cleanup

version 220

fixed collection selection show action lookup for the preview window

files added through the normal import dialog or import folders will be not be fed into the import pipeline if it looks like they are already in use by another process

(hence setting an import folder to a browser download destination should be more reliable)

rewrote some critical phash generation code that was running extremely slow when importing large pngs with transparency

all listctrls that have an edit button now also trigger the edit event on item activate (e.g. double-click/enter key on an item)

added zoom in/out/switch to edit shortcut action dropdowns, can't promise it works everywhere

'stopping' a gallery page search will now correctly make sure the search queue is unpaused so it can move on to any subsequent search

the way excess messages are added to the popup message manager is quicker, improving gui responsivity on message spam

fixed a potential race condition when a file import occurs at the same time as client_files rebalancing

fixed a potential race condition when a check_file_integrity db maintenance occurs at the same time as client_files rebalancing

the client will no longer suppress some core debug stuff

fixed an invalid event handler on canvas close bug

fixed a long-time wx locale issue

fixed a misc main gui parent assert issue

fixed some invalid non-wx-thread calllater calls

cleaned up some inartful showmessage calls

cleaned up some datacache init code

fixed some shutdown event handling

generally improved and fixed in some cases how threads signal job status

improved some thread-interaction timings

improved how files and thumbnails are deleted

file imports add their files to the client_files structure in a more sensible way

misc fixes

misc layout fixes

misc cleanup

version 219

wrote framework for per-mime zoom options--mimes now have separate show actions for the media viewer and the preview window and zoom in to fit

added 'half/double zoom' to limit a mime's zoom to only 25%, 50%, 100%, 200%, 400% and so on (png defaults to this)

added placeholders for zoom in/out algorithm quality, but they don't do anything yet

added a listctrl to edit all this to options->media

added a scrappy zoom values option to options->media

cleaned up a bunch of zoom code

added video/mpeg support

fixed some mime layout stuff

duplicate tag predicates across the 'all known tags' space are merged into more accurate (3-5)-type counts

refactored and simplified how predicate counts are merged

if launched from the manage tags dialog, the advanced content update dialog now correctly filters hta imports to only the files the manage tags dialog was launched with

the advanced content update dialog has a bit of better text to explain this

in prep for a complete rewrite of the image rendering pipeline, merged the fullscreen and preview image caches and cleaned up some related code

added resume recovery to the v215->v216 update code for users for whom this update was interrupted

directory creation is improved across the program

fixed the optimised merge-move file code, which was forgetting to clean up the source when no move was needed

the optmised merge-move directory code will now be much quicker in some situations

all file moves across the program should be less stupid-error prone, and repeating/resuming many maintenance or update tasks that require a lot of moves will be much faster

manual exports, which use copy rather than move, should also be more sensible and faster when repeated

improved database exception rendering, which in some cases was being truncated

misc cleanup

misc small updates to help

version 218

related tags control now updates on new media events (i.e. if the parent media viewer scrolls to different media)

related tags also initialises with suggestions and refreshes the list on media change

related tags now properly filter out dupe suggestions caused by siblings

went over all the gui colours in the client, cleared some redundancies and replaced all the static whites and blacks with appropriate system defaults or chosen custom colours

full size thumbnails can now also be stored in a different location, like resized thumbs, under options->file storage locations

fixed false positive shutdown maintenance checking due to vacuums projected to take longer than the given maintenance time

fixed false positive shutdown maintenance checking due to sqlite_stat1 creeping in

pasting tags or otherwise editing many in at once in the manage tags frame will immediately commit all the changes in one db transaction, rather than one for each tag laggily in turn

removed 'import tags' from file menu--this advanced stuff is now done in and better left to the advanced content update dialog

restoring from an old db backup with a simpler db structure will no longer create conflicts

deleting currently rendering files from the trash should be slightly more reliable

fixed some crash error reporting

misc cleanup

version 217

fixed some high-res video streaming thread scheduling problems with the new video renderer

fixed a cause of huge memory bloat with greatly upscaled videos

to improve seek response time, streaming buffer for the video renderer has a much smaller cap

renderer throttling calculations are more sensible and reliable

the video renderer discards frames to save time if they happen to still be in its buffer

the video scanbar now displays the current frame buffer around the caret!

video canvas now recycles the same frame blit bitmap to save a little time

wrote a prototype related-tags suggestion 'service' for the suggested tags control

you can turn it on and set some options for it at options->tags, feedback would be appreciated

munged increasingly complicated components of the suggested tags control into a clean and proper self-hiding panel

fixed a very important bug that was failing to filter visible thumbnail fetch on mass select and thus massively slowing down the client on large ctrl+a-like operations

open externally button now shows the media's thumbnail, if it has one

open externally and embed buttons now use hand cursor

the simple path tagging dialog panel now cuts off .jpg extensions from filenames on filename parse

if the string component of a generated file export path already ends in the correct .jpg extension, a second will not be added

ipfs unpin will no longer break if the file was already unpinned

the hydrus server now gives filename (for a file save as dialog) correctly on a content-disposition header (this affects the client's local booru as well)

the secondary sort can now be a namespace or rating sort

fixed some potential init problems with some dropdown controls

an edge case object-missing cache retrieval bug is fixed

updated openssl on os x, which might have fixed some problems

updated python on windows, which updated openssl and a bunch of other stuff

updated sqlite on windows

updated linux dev machine to ubuntu 16.04, so a variety of packaged libraries are updated

fixed auto server setup if the client is launched from a windows cmd window

misc cleanup

version 216

video rendering pipeline rewritten to be much smoother

canvas video render timing rewritten to be more frame accurate and smoother

video rendering pipeline will deal with 100% CPU rendering bottlenecks much better

videocontainers will rush to render towards clicked cursor and then slow down

videocontainers will stop rendering as soon as they are replaced on zoom

videocontainers will stop rendering as soon as their parent canvas is closed

added a video buffer size option to options->speed and memory

client_files subfolders are now split into f, t, and r groups for files, full size thumbnails, and resized thumbnails

added an option to override resized thumbnail storage location under options->file storage locations

added secondary sort option to options->sort/collect

fixed a bug where automatically pended tags could exist on top of current tags, forming redundant (1) (+1) situations

fixed a bug when sorting by namespace and a respective namespaced tag has a sibling that de-namespaces it

relatedly, namespace sorting will now filter correctly by collapsed siblings when those sibling pairs' namespaces differ

and collecting will now collect correctly with sibling pairs where the pairs' namespaces differ

simplified how the server caches its account sessions

improved how the server refreshes specific account sessions

the server now refreshes cached session accounts immediately when an admin actions an account type

neatened a little server db account fetching code

fixed file repository petition denial

added a better catch for upnp addmapping error when the mapping already exists as a port forward

the media canvas will no longer accept any mouse or key event if a menu is open anywhere in the program (this circumstance is difficult to pull off, but does cause a crash)

version 215

you can now set different suggested favourite tags for different tag services

added an option to tags option panel to set fixed suggested tags control width

file import status now works on the new window sizing system, and when launched from main gui uses file_import_status frame key

file import status is now a generic apply/cancel dialog when launched from import folder or subscription dialogs

these file import status dialogs are reenabled for linux and os x, as they work properly now!

very large images will render faster and more beautifully on a freshly launched fullscreen media window

all zooming in will be higher quality, particularly for sharper images

better zooming in code also works for very large animations/video

started a framework for better html parsing

updated sankaku downloader to support 'medium' namespace

analyze calls no longer analyze sqlite_stat1, which isn't needed and sometimes gives awkward maintenance timings

fixed server's deleteorphans, which wasn't working with the new thumbnail names

improved speed and weight of server's deleteorphans

fixed a missing foreign key in file_petitions

improved thread worker creation and job processing scheduling

fixed os x media viewer window float_on_parent status

default gui session now defaults to 'last session'

gui recovers from missing default session on boot more gracefully

crash log code improvements now applied to all boot scripts

misc cleanup

version 214

wrote a new resizing dialog 'edit' class for simple resizing dialogs

added comprehensive frame size and position options for the new system to the gui options panel

moved review services frame to the new sizing system

windows that initialise maximised will correctly return to their last remembered size and position on a restore event

maximising a window by dragging it to the top edge of the screen should remember last position as the initial drag start position more reliably

positioning code is a little safer

fixed some missing recalculation of best/min size for wx.notebooks after page change

fixed missing recalculation of scrolledpanel's virtualsize after child wx.notebook's page change

fixed bad parentage for file import status frames

hid file import status button in manage import folders and subscriptions dialogs for non-Windows, as this is very broken, and the parentage fix wasn't enough

improved fuzzy padding on size calculations

refactored and cleaned and harmonised a bunch of the new window resizing code

suggested tags - favourites tag entry in options is now a live autocomplete dropdown

cleaned suggested tags - favourites layout in options

suggested tags listbox now sets its width to exactly fit its tags

improved workflow logic of removing/petitioning siblings and parents (shouldn't get stuck in loops as much now)

tag listboxes will update when tag siblings change

the 'auto-replace siblings' state on manage tags will no longer incorrectly apply to removal actions

import status caches now display errors in a more straightforward way

errors sent to import status caches are now also printed to the log

simplified how database exceptions are caught and displayed

database exceptions now preserve the original exception type

fixed db-side traceback line spacing in database exceptions

improved general database exception rendering

fixed imports for videos with negative start time

deleting videos from the trash that are currently rendering should be more reliable

crash.log now goes to the db dir, unless that isn't determined yet or is unwritable, in which case the traceback goes to console

version 213

created a new tag listbox for the new suggested tags control

added a extremely basic and prototype suggested tags control for favourite tags to options->tags and manage tags dialog

off-screen position detection is now more lenient to account for window managers that position slightly off screen to cut off border

new dialog sizing code adds a little more fuzzy padding on all min size calcs, reducing superfluous scrollbars

hydrus's custom listbook now informs parent scrollable panels of need to recalc scrollbars on page change

the new flexible dialog and frame now catch these events and will expand their size if they need to and have the space

the client options dialog now works on the new sizing system--it will automatically resize when you change pages

some frame resize save event code is improved

the media viewer is no longer aggressive about claiming focus

stacked manage tags panel buttons to allow for thinner dialog

misc dialog cleanup

corrected some bad logic that was excluding transparent gifs from the new thumbnail handling

all gifs and pngs will have their thumbnails regenerated on both the client and server, taking advantage of the recent thumbnail improvements

fixed a bug if you click cancel when choosing an hta path to sync a tag service with on manage tags dialog

fixed a bug where noneditable services (local tags and local booru) could sometimes become editable on manage services dialog. the edit buttons should now disable correctly on init, and there's an additional check in the add code in case of future problems

any superfluous local files/booru services will be deleted on update

version 212

wrote a neat flexible system for recording and restoring a variety of window size and position information

moved the manage tags dialog to the new size and position system

moved the main gui and the media viewer to the new size and position system

the manage tags window launched from a media viewer is now a non-modal frame, allowing interation with the underlying media viewer while floating on top of it

the new manage tags frame commits changes immediately!

review services and file import status will now stay on top of the main gui

fixed some bad advanced content update dialog launch code

the hydrus tag archives in hta syncs can now be anywhere, not just in the client_archives folder

whole bunch of hta sync cleanup

paletted images are now dequantized to RGB(A) before thumbnail generation, enhancing scale quality

'LA' (greyscale+transparency) images will now render with correct transparency

png files will now always generate png thumbnails (previously it was just ones with transparency)

trivial options changes are less frequently saved

rule34@booru.org now parses namespaced tags

fixed a bug in thumbnail resize error recovery code

fixed a bug in file repository petitions

updated to Pillow 3.2.0

misc cleanup

version 211

added options for disk cache init and maintenance to 'speed and memory' page of options

abstracted the manage tags dialog into a general purpose 'apply' dialog and a scrollable manage tags panel

the 'auto-replace entered siblings' checkbox now remembers how it was last set

uncertain sibling tag counts will now appear like 'tag (1,234-1,456)', showing the range of possible results they represent

fixed a bug where siblings would be collapsed inside the tag manager object if a specific tag search domain was set

altering the search text on an autocomplete from 'tag' to '-tag' or vice versa should graphically update more reliably, and an exact-matching predicate will be promoted to the top of the list correctly either way

calls to upnpc will now kill the process and raise an exception if it hangs for more than thirty seconds

misc improvements and cleanup

version 210

manage tags dialog now launches much faster when thousands of thumbs are selected

manage tags dialog now has a prototype 'do this for all' checkbox to make mass selected-tag editing a bit quicker

polished the new disk cache turbo mode, added free memory limits and more options

a bit of disk cache population now runs on client boot, just to kick things off

a full disk cache population now runs before repo sync

the client now keeps about 200MB of itself in the disk cache during idle maintenance

the update files in client_updates are now stored compressed

the log files and 'already running' files are now stored in the db folder, meaning hydrus shouldn't have to write to anything but db dir and temp dir

'already running' files now delete themselves on program close if they can

the log directory is no longer needed

if there is only one booru to select, it will be auto-selected

client failed exit is less likely to hang, more likely to log error

client won't bother to save lest session regularly or on exit if this is not the default startup session

hotfixed an auto-vacuum bug

added a catch for any future vacuum mess-ups to save the db from collapse

added other catches for other close/reinit db calls

misc cleanup

version 209

the advanced content update dialog can now be !aunched from manage tags, and will only affect the files the dialog was launched for

gave the advanced content update dialog a bit of a makeover

system:ratio now supports 'taller than' and 'wider than' operators

merged client thumbnails into the dynamic file directory structure

the client files manager now manages thumbnail paths exclusively

thumbnail import responsibility moved to the client files manager

thumbnail resizing responsibility moved to the client files manager

orphan clearing and thumbnail regeneration maintenance tasks moved to client files manager

'regenerate thumbnails' now has an 'only do missing' mode

thumbnail regen on resize is faster

copy file to clipboard responsibility moved to the client files manager

media file and thumbnail deletion responsibility moved to the client files manager

if a full size thumbnail is missing but regenerated ok, the client files manager will now only produce one, better error message

cleaned some bad error detection and reporting code

cleaned some regular 'thumbnail size has changed' code

fixed and cleaned a mix of file and thumbnail path code

client files rebalancing and recovery will now efficiently move-merge rather than copy-merge

fixed some bad portable->absolute path normalisation

improved missing external storage location error reporting

server thumbnails are similarly merged into server_files

to keep things simple, the client no longer keeps track of number of local vs. remote thumbnails for file repositories

perceptual hashes are now calculated from original file, not thumbnail, and only for jpegs and pngs (videos had too many black frames etc...)

the vacuum maintenance popup will now only appear if a vacuum can be completed in the time available

slideshows will now yield to any open popup menu, not just those launched from their own canvas

wrote a prototype turbo mode and put it on the debug menu

misc cleanup

version 208

split the path tagging dialog's complex widgets into simple/advanced panels

added filename and directory 1, 2, and 3 easy tagging, with namespaces to path tagging dialog

fixed up some buggy 'tags just for selected files' behaviour

ipfs directories now support a note

the ipfs directory note can be edited from review services

added 'all namespaced tags' and 'all unnamespaced tags' to the advanced content update/service-wide operation dialog

added 'clear trash' button to trash panel on review services

separate timestamps for file repo upload and ipfs pin will be displayed on single-thumb right-click menu

the correct file domain timestamp will be used for newest/oldest sort

db integrity check now scans all attached database files

broke up mappings petitions in the server db into more efficient 'deleted' and 'petitioned' parts

cleared out some static refs in prep for external db location

simplified a bunch of file mirroring code and improved error handling of directory mirroring

fixed a rare thumbnail resolution calculation bug

misc smaller fixes and cleanup

version 207

finished ipfs directory creation

ipfs review services pane now shows current directory shares

cleaned up some content event responsibility in ipfs service code

fixed a file download bug when using ipfs 0.4.2

wrote some more ipfs help, for directories

popup messages can now hold arbitrary copy-to-clipboard text content

fixed the new listbook changes for Linux, which was having trouble tracking the client_data inside the sorted listbox

fixed an issue with the custom filter management dialog possibly duping the 'default' entry on ok, making impossible to again launch

offloaded tag parent generation responsibility in the a/c write tag pipeline

added a checkbox to manage tags dialog that will control if siblings are auto-replaced on entry

all the clunky 'add preferred sibling' workflow is removed from the manage tags dialog

fixed incidence tag sort order not initialising properly in the options dialog (and transferring to lexi (namespace) every time)

fixed too-eager quit during shutdown db analyze

fixed the db restorebackup call, which wasn't closing the gui correctly

the manage services dialog will now automatically re-check 'allow zero' on numerical ratings with only one star

misc cleanup

version 206

if the currently focused thumbnail is removed from view, the earliest non-selected thumbnail will be remembered as a 'ghost' focus if the user then presses the arrow keys to continue to navigate

if the user presses an arrow key to navigate the thumbs when there is no known focus thumb, the first will be assumed

the listbook now supports duplicate display names

services of the same type can now have the same name

cleaned some listbook code

inbox or custom filters will now render first-file animations correctly on initialisation

fixed the 'change' event on the new timedeltabutton, which was not updating the thread checker on dialog ok

subscriptions will now naturally terminate their gallery sync on a 404, rather than dumping out with an error

fixed some false positive 'paths are different' testing in default directory mirroring code (used in db backup, restore) that was slowing these operations down

if database files have the same file and modified timestamp, they won't be re-copied in a backup or restore

client external storage location recovery/rebalancing will now skip over files with the same size and modified date

if you backup the client and have external file storage locations, a popup will remind you to back those locations up manually

did a quick pass over some help stuff, and added a 'can the client manage files from their origial locations' bit to the faq

created a simple no-reward patreon, wrote some help about it, and added links in the usual places

empty 'read' a/c dropdowns will be more careful about idly refreshing their system preds, hopefully leading to less accidental gui hang on large jobs

main gui menus will be more careful about refreshing themselves when called, hopefully leading to less accidental gui hang on large jobs

text snippets are stored inside the client db in a better way

added flat service directory structure storage support to the client db

fixed a mapping petition rescind sql typo

fixed the v198->v199 server update code

misc cleanup

version 205

fixed v201->v202 update code, which v204 retroactively broke

wrote a new maintenance routine to regenerate the ac cache if it ends up miscounting or not generating correctly

fixed specific ac_cache generation code, which was double-counting because of recent file-add optimisation

client db maintenance now vacuums attached databases according to their own vacuum clocks

client will more aggressively vacuum and analyze if the client has been idle for an hour

client vacuum now prints success statements, with duration, to the log

analyze now produces an auto-dismissing vacuum-like popup message

the 'is maintenance due' check on shutdown now accounts for the new vacuum logic and includes analyze needs as well

shutdown maintenance will use more of its allotted time if needed

tag archive sync now always filters to local files on initial sync

if you import a tag archive with sha256 hash, you are now given the option to filter to local files or import everything

the occasionally crashtastic opencv.imread call is now completely replaced with PIL, let's see if it's too slow

the new-but-now-redundant 'disable opencv for static images' option is removed as a result

fixed a redirection location parsing bug that was affecting sankaku subs with >1000 initial files

fixed some bad analyze timestamp tracking code

fixed advanced content update code, which still had some mapping service_ids floating around

version 204

current, deleted, pending, and petitioned mappings are now stored on service-separated dynamic tables

any tag service deletion or reset should now only take a few seconds

fixed a physical file deletion bug that seems to have affected (some?) versions of linux and os x

reintroduced a revamped 'clear orphans' database->maintenance routine for the client, which will delete or move orphaned files and delete orphaned thumbnails. run this if you are on os x or linux

improved misc file deletion code

fixed some multiplatform path conversion, and generally improved how paths are stored and retrieved

the network application of a parent tag to all files with the child tag now happens serverside, and only on petitioner-approval of a tag parent relationship. it still occurs locally to the uploader, but is now wrapped up in the parent commit

fixed some numtags search logic

fixed an issue when mixing system:age with any tag-based predicate

media_results db calls no longer require a file domain

'hide inbox and archive preds if either has no files' now defaults to off

added an option to disable OpenCV for gif rendering

the selection tags panel now initialises with the correct tag domain (it was previously always starting as 'all known tags', which is not always true for session pages)

version 203

thumbnail resize now happens on the fly--feel free to change it as often as you like, it takes no time at all

added 'lexicographic (grouped by namespace)' tag sorting to the selection tags box

added an option to disable OpenCV for static images under the media options page

panning with the shortcut keys now pans by a twelfth of the media size or canvas size, whichever is smaller

cleared the analyze timestamp cache for both client and server, which will force a reanalyze of the new db files on the next db maintenance run

improved some misc analyze code--primary keys are analyzed again

added an explicit analyze command to the client's database maintenance menu

the server now analyzes attached database files

improved an unhelpful server thumbnail error message

added an invalid NULL check-and-skip to hash cross-referencing

fixed some invalid a/c write dropdown search domain initialisation

fixed some borked zoom calculation code that was sometimes lagging the media viewer and leading to 100% zoomed images being sent unneccessarily and then being nastily scaled down

fixed a file import error if a synced tag archive is missing

updated sqlite for windows

misc cleanup

version 202

fixed a problem with the v198->v199 update step

added a bad error catch in the vacuum step in v200->v201

added some rollback exception handling for unusual situations

the autocomplete caches under the client_cache subdirectory are dynamically folded into a single client.caches.db file, and that file is folded into the main db journal like the other attached dbs

ac_cache file and tag processing lag and edge-case autocomplete miscounting should be removed as a result

cache generation is optimised for empty services

specific file_caches' add and delete file commands now modify their tables directly, saving a whole load of time that was previously done superfluously filtering

the specific ac_caches' add mappings command is a bit more logically efficient

increased cache size for each attached database

folded all 'attached' db update code into the main update_db routine

folded all 'attached' db init code into the main create_db routine

made the vacuum checking routine safer

reduced analyze log spam

analyze maintenance breaks its larger jobs up better

analyze mainenance now analyzes external databases

the db connection now refreshes every half hour, to regularly clear out journal files

the hydrus db assumes it generally has exclusive control over its db, so it no longer wraps its read-only requests in transactions

session load now happens off the main gui thread, and media generation is broken into 256-file chunks, which reduces startup db and gui lag for large sessions

the youtube video downloader now lists a webm option if youtube provides one (they seem to offer 360p webms on everything(?) now)

the timedeltacontrol (where you set the period for the thread checker and import folders and so on) is replaced by the more compact timedeltabutton, which will launch a dialog with the old control when clicked

the server will no longer revisit old updates and cull since-deleted content, as this was proving cpu-burdensome and not helpful to future plans for update dissemination, which will rely on static update files

misc cleanup of some server db stuff

content update processing will not spam menu updates throughout, but only notify once at the end, which should reduce idle gui hang due to db access choke

the autocomplete tag entry will not refresh system pred menu during idle, which should reduce idle gui hang due to db access choke

shutdown repo sync will now report the update summary text and will not spam the intervening statuses to the log

moved timestamp out of the serviceless media_result and into the locations_manager, to reflect the database's knowledge of each current service having a different timestamp

local and trash timestamps are generated from the locations_manager now, and non-local file service timestamps will soon follow (e.g. 'pinned to ipfs_service 3 days ago')

if the timestamp is unknown for the current service context, it will not be displayed

file repositories will now only sync thumbnails when their updates are completely synced (to stop 404s from since-deleted files)

fixed a ( 0, 0 ) resize event bug that was sometimes causing borked media containers on media viewer shutdown

syncing to a sha256 tag archive will still import the data as requested, but a popup note will explain that as everything will be imported, further syncing is pointless., and the sync will not be saved

fixed a bug in hta export if you click cancel on the hash_type choosing dialogs

misc cleanup

version 201

exported hash and tag master tables to external database files for both client and server

the client's 'mappings' table is split into 'current_mappings' and 'pending_mappings', saving about 20% of space and speed for almost all mappings operations

removed superfluous rowid column from client mappings tables, saving about another 20% of space

removed superfluous rowid column from server mappings tables, saving about 20% of space

to save time, service mapping counts now only aggregate current mappings, not current and pending

exporting to tag archive now only exports current_mappings, not pending

'all known files' queries will more cleverly discern their current/pending search domain

tag count queries will more cleverly discern their current/pending search domain

cleaned up a lot of mapping-related query code, and any ugly status leftovers from current/pending union

num_namespaces is no longer tracked for tag services

unified clientside service deletion code, fixing several orphan-generation issues

db update will remove all mappings orphans created by a recent tag service deletion

some db settings init is harmonised into one location

fixed up some external db settings init

db connection settings are now applied to all attached db files

db vacuum code is harmonised into one location

vacuum cleans up after itself better

db vacuum is now preceded by a hard drive free space check

db vacuum can now predict how long it will take

if a vacuum fails for an unexpected reason, the db will now recover better

the db recovers from failed backups better

the server db will vacuum all its databases on a backup, except any that it thinks will take total vacuum time to more than five minutes

server backup will report roughly how long it took, clientside

significantly sped up system-tag-only 'all known files' queries for tag services that have many mappings

significantly sped up system:numtags for =0 or >0 when tag services have many mappings

the status bar has some improved grammar, reporting overall common mime as appropriate

the status bar reports single-thumbnail info text when one file is selected

fixed an image cache key type-matching bug

data cache now tracks objects with dynamic memory usage

the static image rendering-cache relationship has been slightly dejanked

added a couple of checkboxes to the options->gui page to hide/show thumbnail title banner and page indicator

the manage tags censorship dialog now defaults to the 'all known tags' service on init

slideshows will not progress if the current media is either a static image that has yet to render or an unpaused animation that has yet to run through once

service content package status messages now summarise their final commit status better

this nice new line is printed to the log as a record of repo sync rows/s speed

the client will shut down faster if big jobs are running

version 200

added 'censor tag' to tag right-click menu

cleaned tag censorship a bit and moved from the gui down to the db layer

fixed tag censorship for multiple services

sped up tag censorship a bit

improved some tag censorship logic for unusual namespace situations

tag censorship now applies to siblings and parents, even in their management dialogs!

fixed the missing fit zoom when scrolling through zooms and when flipping between 100% and fit zoom

cleaned some canvas init code, which was loading bad initial zoom values due to other recent changes

restored standard static image zoom mode for cv-supported images to high, inter_area quality

cleaned up some image/animation container init code

cleaned up some image cache image tracking code

fixed ipfs pin, which was conflicting with the new ac caches

extracted client deleted_mappings to their own db table

prepped mappings processing to work with status-separate mappings tables--it is also a lot cleaner now

because of the deleted_mappings extraction, a bunch of mappings membership tests are much simpler now

upped the db cache size to a semi-ludicrous 150MB, which nonetheless seems to improve processing speed

improved how hydrus databases manage their db file locations

improved how the client and server databases backup/restore

improved some non-local thumbnail fetching code

converted some old YAML stuff to JSON

improved big job polite pausing code

cleaned up some clientside imports, and set the client to only import its server code if it needs to boot a local service

when an import folder needs to rename a file it is about to move because of a conflict, it will now add autoincrementing numbers, starting at 0, and before the file extension

added some missing shortcut actions (pan_x and set_media_focus) to the shortcut input entry dialog

tags and siblings that match the current WRITE autocomplete tags entry will no longer have doubled parents!

improved how system:limit is applied to searches

version 199

added 'all known files' autocomplete caches, so manage tags dialogs' autocompletes should now always appear quickly

the now caches will be created on startup, which may take a few minutes

deleted the old tag autocomplete cache

wrote a faster initialisation routine for all known files caches

fixed an ac_cache cleanup typo

the client's (usually gigantic) mappings table is temporarily extracted to an external db file, which should have numerous benefits and perhaps a few drawbacks for now

the update will take a few more minutes, and will need lots of disk space, to extract the mappings

the server has a very similar external mappings table, so it will also need a while to update, but future admin backup calls will be a lot quicker, due to a much smaller vacuum

greatly optimised a mass-membership calculation in tag processing

reduced some db check-spam timer that was adding up idle cpu time because of recent ac_cache additions

fixed predicate parent sorting

if swfrender fails to generate a swf thumbnail within 60 seconds, the client will now dump out (rather than hanging indefinitely)

fixed some string-selection dialog layout rules

optimised some 'all known files' search queries

subscriptions will now append new urls in oldest->newest order, roughly preserving that order in the internal url cache

the 8chan thread watcher now accepts .json API urls directly

the repository processing routine will handle server_busy errors more gracefully

cleaned and refactored most media viewer 'canvas' code, dejankifying a whole bunch of related stuff and generally improving the object hierarchy

cleaned up and unified a lot of logic and responsibility for idle checking and shutdown maintenance stop time for some big jobs

misc cleanup

misc dialog cleanup

version 198

ipfs download now parses directory multihashes

ipfs directory trees are thrown up on a new checkbox tree dialog for selection/filtering

created a new download_urls (i.e. multiple urls in one popup) popup call, which the new ipfs directory downloader uses

improved a very inefficient line of sql in the mappings fetch stage of query result building that was also used in the manage tags dialog. on my dev machine, fetching 256 files' mappings dropped from 2.5s to 9ms!

regular gallery pages and 'page of images' pages now have a 'paste input' button to mass-add newline-separated queries from the clipboard

fixed a typo in new txt_tags import folder code

fixed bad autocomplete predicate sorting

fixed some sibling autocomplete search_text matching logic

some byte-based value/range presentation will now correctly convert to bytes, rather than regular decimal numbers

improved linesep splitting code across the program

fixed get/setlabel calls across the program to get/setlabeltext, so ampersand characters are handled properly

wrote an 'all known files' autocomplete cache db and prepped a bunch of code for it--it should be easy to finish next week

added a new regex practise link to the regex dialogs

os x app release should display better on retina displays

misc cleanup

version 197

on client boot, autocomplete caches for specific file/tag service cross-references are now initialised and populated. progress is shown on the splash window

on client boot, surplus autocomplete caches are deleted

on service add, new autocomplete caches are created

on file add/delete, autocomplete caches are updated

on mappings pend/add/rescind pend/delete, autocomplete caches are updated

the new autocomplete caches are consulted for all non-'all known tags' queries

the old autocomplete cache no longer stores counts for specific file services, and the remaining associated maintenance calls are deleted

databases now start their own mainloops

databases now wait for their mainloops to finish prepping any large caches before they return to the controller

the client database waits for autocomplete caches to finish before it finishes its own mainloop

the padding around flash and the animation bar are included more accurately in some media zoom calculations, which should eliminate some general zoom jankiness and accidental 100% flash zoom coincidences that filled up the whole canvas

fixed some clientside server boot error spam when local server or booru had no port set

account refreshes that fail due to a network error will spam less to the log

fixed .txt unicode tag parsing from import folders, which was not decoding at the correct step

administrator immediate repository syncs now sync thumbnail downloads if needed

service thumbnail sync will no longer superfluously check the presence of thumbnails whose files are local

if a tag entered into the manage tags dialog has a sibling that already exists for all files, then a new 'ignore, as the sibling already exists' choice will appear

fixed an overcounting bug in 'selection tags' when importing and adding tags at the same time

fixed a typo in repository sync status text that was overcounting total number of updates by one

fixed youtube downloader, which broke with the new library on my new dev machine

the way that tags and predicates are filtered against a tag autocomplete text entry is now much faster

bumped up the default content update chunk threshold from 100 rows to 5,000, which seems to be speeding up processing significantly, with a cost to recovery latency--see how it works for you

version 196

fixed the 8chan thread watcher for boards that host content on media.8ch.net

improved the thread watcher url check logic so it won't lag with the new fix

cleaned up the ac generation code a little

'all known tags' ac counts are now summed from all the known tag services rather than calculated directly (a <= indicator for when these cases overlap will be forthcoming). this speeds up file add/delete, service reset, a/c fetch time, and general tag processing, and reduces the size of the db

ac generation code now deals with 'is the entry text an exact match or not?' better

ac generation code will now no longer produce non-exact-match siblings on an exact match search

ac generation code will no longer save half complete search text into the db as new tags

on update, the a/c cache and its helper table 'existing tags' will be cleaned of a lot of orphans, which may take a few minutes

fixed some bad unicode path parsing when importing files in some OSes, I think!

fixed some bad read autocomplete sibling substitution

fixed a bug where autocomplete predicate lists would not update if the new list was merely a reorder (which can happen in some unusual sibling cases)

fixed the tumblr parser for the subtly new API

import folders now support loading tags from neighbouring .txt files--check the dialog to set up which tag services you would like to import to

the ipfs file downloader now queries DAG object links, determines if the given multihash is a directory or other complicated object, and if so politely dumps out (handling of directory downloads is forthcoming)

some db code is cleaned up

prepared db code for some future subclasses

wrote most of the new ac cache db

misc cleanup

added some browser addon links to the ipfs help

version 195

wrote up a v1.0 ipfs help page

added ipfs services to 'system:file service'

added a 'multihash prefix' option to ipfs services in 'services->manage services'--it will prefix the given text to multihashes copied to the clipboard (e.g. you could use 'http://127.0.0.1:8080/ipfs/')

fixed a bad repo sync processing bug that was incorrectly assuming packages were completely processed when they often were not

all repos will have their processing timestamps reset on update, forcing a (fast) reprocess of everything they have to cover the previously missed content

fixed 8chan OP image parsing, which was sometimes failing due to absent md5s in the json

widened the splash screen a little more so repo sync text can fit a bit better

a couple of splash status texts are shuffled around

updated to sqlite 3.11.0 for windows

if sqlite3 reports at least version 3.11.0, the db will stick to WAL for large transactions (lke vacuum and content processing), since these are fast now

'namespace:*anything*' will now only appear if the a/c input has no 'subtag' component., e.g. 'character:'

import and export folders now print simple summaries to the log if they do some work

tidied up the 'pages' menu

the various hardcoded 'delete' key events are now also triggered by a backspace on os x

added a rough 'copy known urls' to the regular thumbnail menu. this is prototype, let me know how you would like this information managed and displayed in future

in prep for the new cache layer, the autocomplete dropdowns (and hence search domains of all contexts) no longer support 'all known files' cross referenced with 'all known tags'

misc cleanup

version 194

ipfs pins and unpins can now be queued up like file repository pending and petitioned, through the regular thumbnail right-click menu, which also reports some/all ipfs pinned selection status

this ipfs action queue is similarly summarised and commited at the normal service 'pending' menu

ipfs's 'pinned', 'to pin', and 'to unpin' statuses are displayed on thumbnails with ipfs-specific icons

you can copy the focussed file's ipfs multihash or all the selected files' ipfs multihashes from the thumbnail menu's share->copy->ipfs multihash

added a .txt tag parser to the 'path tagging' import dialog--it will parse the same sort of txt files the export dialog produces

the client's new 'requests' network code is harmonised, generally improved, and now produces hydrus-compatible exceptions

updated help re the local server and boorus now defaulting to off

db can now remember service-specific filenames (e.g. ipfs multihashes)

cleaned up some overly complicated and confused thumbnail menu code

the pending menu now specifies what it is about to do more plainly

version 193

the client's local server and local booru can be turned off from their respective management panels, and from now on, the client will initialise with them this way.

if the local server or the local booru are not running, their copy/share commands won't appear in right-click menus

the welcome dialog is now a simpler popup message

incidence sorted tag lists are now sub-sorted by a-z lexicographic

pasting many tags that have siblings to the manage tags dialog will ask you if you want to always preference the sibling, saving time

added a 'clear deleted file records' button to the local file service on the review services window

idle mode now cannot naturally engage within the first two minutes since client boot

the autocomplete search logic will not count namespace characters in the autocomplete character threshold, so typing 'character:a' will not typically trigger a (very laggy) full search

putting a '*' anywhere in an autocomplete search_text will force a full search, ignoring the a/c character threshold

moved some specific 'give gui time to catch up' pause code to the generalised pause/cancel code that a lot of stuff uses, so big jobs should generally be a bit more polite

split the daemon class into two--one for big jobs that remains polite, and another for small jobs that triggers regardless of what else is going on. this should increase responsivity for a number of scenarios

fixed some bad wal failure detection and hence no-wal file creation on some instances of db cursor reinit (usually after service modification). because of now many superfluous no-wal files, existing no-wal files will be deleted on db update

some external storage location errors are improved

some internal and external storage location init is improved.

if an error is detected in the external storage location manager, it will not attempt to rebalance again until the client is rebooted

improved some upnp error catching

cleaned up some misc shutdown thread-gui interaction error spam

did some prep work on a future rewrite of daemon jobs pipeline

split up some mixed file/data/404 'stuff was missing' exception code

version 192

added a 'check on ok' button to the manage subscriptions dialog's subscription panel

check file integrity now prints missing paths to the log

fixed a typo that was breaking some repo sync download resumes

if an external storage location does not exist on client boot, a popup will say so

resetting a service is now much faster when the service is large

some repo processing stuff is a bit faster

improved some image rendering error handling

some thumbnail errors are recovered from in a better way

improved db update bit rot resilience

improved the vacuum disk space error popup with better explanation of the problem and solution

misc improvements

moved to win 10 dev machine, so several windows libraries are a bit newer

updated to opencv 3.1.0 on windows

version 191

added ipfs service type

added simple ipfs service gui to manage and review services windows

added simple ipfs download and import to pages menu

split confused service data object into a bunch of more flexible and readable subclasses

raw url downloads are now powered by 'requests' rather than my hacked http engine, and work a hell of a lot better. more to follow

raw url downloads now pause and cancel correctly

optimised some a/c cache clearing in mapping update processing

optimised content update object -> db yield logistics

added no-daemons command line switch to disable all daemons

added no-wal command line switch to disable WAL journalling for the db

if the db fails to read a new WAL-journalled db file it has just created, it will delete the file and attempt to recreate the db, never trying WAL again

changed the way the db writes big repo syncs to disk so that the job can be abandoned much more quickly

improved local file not found error to state the directory or exact path that was the problem

fixed some socket-level error reporting (I think timeout was part of this)

the petition approval and denial client-to-server network commit is broken up into smaller steps to reduce server lag and the overall likelihood of timeouts

subscriptions will no longer add files to their 'successful files' button when the file's url seemed new but it actually was previously deleted

subscriptions will now identify themselves with an additional popup message when they encounter critical errors during sync

editing a clientside service will force a reset of the appropriate session cache, so if you change access key, the account changes will be reflected immediately

the import files dialog will no longer spam error popups on uninteresting or empty files--its 'parsing complete' statement is more verbose instead

the 1.5s gap between parsing complete and the buttons being enabled is also removed. this was supposed to be helpful, to give you a moment to check the files were as expected, but in the end it was just annoying

fixed a typo in some thumbnail error catching and regeneration

improved my network code syncing, so linux and os x releases are cleared out of many old files

misc code cleanup

version 190

fixed some hashing recalculation in collections that meant they reported no files in many cases

fixed some hashing recalculation when 'remove files' is called

improved the way the client db stores file information and service->file mappings

idle processing jobs will now explicitly wake up as soon as the client naturally switches from not idle to idle

the minimum allowed value for the 'max cpu %' control in the maintenance and processing options panel is now 5%

the maintenance and processing panel is rewritten and laid out a little clearer

'busy' is now 'system busy' on the status bar

force idle and force unbusy are now merged into a new 'force idle' that sticks until you explicitly turn it off

busy and idle states should now update immidiately after closing the manage options dialog

improved exit code event order to be less rude to the OS

improved exit code emergency event handling

fixed a typo that was not appropriately skipping the 'do you want to run shutdown jobs?' dialog

file storage folder rebalancing will now occur on shutdown maintenance

the client now closes down more reliably if the db fails to boot

the client now closes down more reliably if the gui fails to boot

if a client vacuum fails, it'll now also raise the exact error sqlite gave

fixed ctrl+a on the autocomplete dropdown (I think the Insert/IME support change broke it)

the ways the 'read' tag autocomplete control talks to other gui elements is improved

the tag autocompletes will now refresh their results lists on more internal variable changes

the query page management controller manages fewer redundant variables

updated sqlite for windows

the client and server dbs will attempt to change to TRUNCATE journal mode if WAL causes a read disk i/o error

misc code cleanup

version 189

split the big analyze db calls into individual table/index calls and moved them from update code to the normal maintenance routines

on vacuum, both the client and server dbs will now bump their page size up to 4096 if they are running on windows (server vacuum is triggered by running a backup)

vacuum should be a slightly faster operation for both the client and server

boosted the db cache significantly--we'll see if it makes much difference for things.

the way the selection tags control updates its taglist on increases to its media cache is massively sped up. An update on a 5,000-thumbnail-strong page now typically works in 3ms instead of ~250ms. large import pages should stream new results much more quickly now

sped up some slow hash calculation code that was lagging a variety of large operations

some hash caching responsibility is moved about to make it available for the add_media_results comparison, which now typically works in sub-millisecond time (was about 16ms before)

some sorted media list index recalculation now works faster

some internal media object hashing is now cached, so sorted list index regeneration is a bit faster

some medialist file counting is now superfast

wrote a new pauser object to break big jobs up more conveniently and reduce gui choking

the repo processing db call now uses this pauser

some copy and mirror directory functions now use this pauser

backup and restore code for the client now skips re-copying files if they share the same last modified date and file size

backup code for the server now skips re-copying files if they share the same last modified date and file size

http cannotsendrequest and badstatusline errors will now provoke two reattempts before being raised

socket error 10013 (no access permission, usually due to a firewall) is caught and a nicer error produced

socket error 10054 (remote host reset connection) is caught, and the connection is reattempted twice before being raised

the old giphy API is gone, so I have removed giphy

forced shutdown due to system exit/logoff is handled better

pubsub-related shutdown exceptions are now caught and silenced

an unusual shutdown exception is now caught

fixed a copy subtag menu typo

cleaned some misc hydrus path code

tags that begin with a colon (like ':)' ) will now render correctly in the media canvas background

some misc code cleanup

dropped flvlib since ffmpeg parses flv metadata better anyway

version 188

if you have custom filters set up, they will now be listed in the normal thumbnail right-click menu, which will expand for them

'refresh account' button's event no longer waits for service response on the gui thread

'refresh account' button on review services will now re-enable if the call fails

added 'hide inbox and archive predicates' checkbox to options->default file system predicates. not sure if this is the best place for it!

system:hash can now query md5, sha1, and sha256

existing pages with system:hash will break on update. since this is such a rare predicate, I assume this is not a problem!

improved db hash cross-reference flexibility

any pausable popups will now explicitly change their text to 'paused' when they are paused

upload pending popup now has a title, so it should pause a little more sanely

regenerate thumbnails popup now has a title, so it should pause a little more sanely

cleaned up some job pause status logic

upload pending now works on a stream, so gigantic uploads will cause fewer hitches and overall problems

upload pending code is cleaned and simplified in several ways, with better error handling and progress reporting

synctotagarchive is now split into a thousand tiny separate jobs and runs on a separate non-db thread. it will no longer lock the db the whole time

synctotagarchive popup should be a lot more responsive and will specifically give the gui some time to catch up after every chunk of data processed

synctotagarchive will update pending tags count (and gui stuff will show the pending tags) as it goes

synctotagarchive is now a pausable and cancellable operation!

large repository sync jobs should be a little less laggy on final commit

system:duration now displays its value as a properly formatted milliseconds time delta, rather than just a big int

profiles now flush to the log immediately

some cpu-heavy daemon recheck periods have been extended to reduce potential hangs on slower computers

some repository sync messaging wording and logic is improved (it should report what it is doing more reliably when you are cpu-busy)

reordered instances of 'waiting politely', so download text/gauge updates should occur before the wait does

some misc 'waiting politely' logic is improved

pixiv ugoira is detected and a better error (that will be ignored by the new sub error throttling code, for instance) is produced

'uninteresting mime' errors have nicer import status note text

fixed the 'thumbnail grid background' custom colour option

version 187

file->restart now works for linux

the 'call to' thread pool now has a delayed start, which fixes a shutdown hang in linux (and perhaps the other platforms, in one way or another), when the user hits 'forget it' on the 'already running' boot dialog

removed some console error junk on abandoned client boot

if a requested full-size thumbnail is missing, the client will now attempt to regenerate it

if a requested full-size thumbnail does not render (for instance, during a request to generate a resized thumbnail), the client will now attempt to regenerate it

some thumbnail fetching and generation logic is slightly improved

you can now drag animated media around when the inital drag click begins over the animation canvas

ditched some ultimately unhelpful mouse warping when the cursor was near the edge of the media viewer when a drag began

possibly fixed a strange tags_fts4 mirroring problem that has hit some users for a reason I do not understand

import tag options panels will no longer order their tag services randomly

fixed a bug where import folders would not save changes to their explicit tags

fixed a bug where under some circumstances, the gallery importer was attempting to display previously deleted files on the screen, causing errors.

subscriptions will now pause a few seconds after a file import error, which should solve a problem with client hanging when lots and lots of subscription file errors occur one after another

if subscriptions encounter 5 file import errors, they will pause their sync and try continuing later

pixiv manga error handling is improved

bad mime error handling is improved for all importers

fixed an important bug in the import status cache where error texts were not being stored as unicode, so error texts with non-unicode-compatible characters were causing exceptions when the session was being saved to the db or the import status frame was being loaded

existing sessions' import status caches should automatically convert their 'notes' to unicode on update

removed booru db fetch spam when booru download page has no work to do

fixed a typo in regenerate thumbnails popup message

fixed some bad event handling for the gallery page's file limit control

updated some regex defaults to have better multiplatform support and handle a wider array of valid characters

version 186

Windows release is now 64-bit!

some libraries got updated for windows, to varying minor effects

BeautifulSoup updated to latest version for everything

fixed a warning from newer versions of BeautifulSoup by explicitly using lxml

updated the running_from_source help file to what is currently needed

ffmpeg.exe is also 64-bit

wrote a "reducing program lag" page for the help

added program restart to file menu, doesn't work for linux yet

improved restart code (it was used in restore database, and for some users would not close the parent process down properly)

in the media viewer, page up/down, arrow keys and mouse scroll no longer unhide the mouse

in the media viewer under Windows, dragging no longer unhides the mouse

the implicit delay in the downloader-importers is reduced, so redundant files should always stream in a super fast now

export files dialog will now auto-create missing destination directories

export files dialog will now work off the gui thread (to stop gui hangups on big jobs) and will report its export status through the export button, which will be disabled until the job is finished

changed the new tab/shift+tab shortcuts because they break panel control navigation (I want to make these editable soon):

'fetch autocomplete results now' shortcut is now Ctrl+Space (Raw Control for OS X, not command)

'IME mode toggle' is now Insert

cleaned up how control is handled cross-platform--in general, 'control' is always control for windows and linux, and 'control' is command for os x. this may be incorrect.

json dump errors now report more information about the specific object and its serialisable contents that are causing the problem

clipped sibling/parent reason entry dialog message when lots of pairs are added

fixed a bit of bad logic in sibling/parent dialog->content-update workflow

fixed the thumbnail canvas not refreshing on a canvas shrink caused by expansion of the management panel that doesn't change the thumb column count

fixed emergency boot error reporting for when HydrusData won't load

added some unicode conversion unit tests

version 185

right clicking on a tag/predicate in the 'selection tags' box or the 'search' list of active predicates provides new intelligent menu options to discard/require/permit/exclude the selected tags from the current search. try it out!

this system also works for namespace and wildcard and system predicates

'predicate a' and its inverse '-predicate a' are no longer considered the same!

namespace and wildcard predicates will now remove their inverts if they are added to the list of active predicates

the way inverse predicates are generated and compared is generalised and improved

fixed 'open a new search page for tag x', which might have been broken?

quick-entering a tag in the 'read' tag autocomplete entry will now always replace with a sibling if one exists

quick-entering a tag in the 'write' tag autocomplete entry will now not replace with a sibling--you will get the option of what you want to do

when this happens, the preferred sibling is labelled on the 'what do you want to do?' popup in the manage tags dialog

also when this happens, the affected file count for add/pend the preferred sibling should now be correct

fixed client support for tags that begin with ':', such as many emoticons. these should now be parsed correctly from websites and keyboard input, keep their leading colon through network and db conversions, and display correctly (for the most part!), despite a little bit of magic to make them work

created a new TimeDelta control to handle finer tuning of optional periods

improved timedelta->pretty string code to support >60s times

manage subscriptions dialog now supports day and hour period, with min period of four hours

manage export folders dialog now supports day/hour/minute period, and the parent dialog displays that time delta appropriately

manage import folders dialog now supports day/hour/minute period, and the parent dialog displays that time delta appropriately

the thread watcher now supports hour/minute/seconds check period

folded 'search' and 'download' menus into 'view' and renamed it 'pages'

'sessions' moved from 'file' to the new 'pages'

moved 'admin' menu to 'services->administrate services'

added 'forced system:limit' to options->speed and memory panel

if several system:limits exist in a search, the minimum will be used

added a checkbox to options->downloading to replace the traffic light waiting politely indicator with text

subscriptions with failed files will say 'x urls in cache, y failed' on their dialog panel

at the gui level, media that superfluously receives a 'pend tag' instruction for a tag that is already current will discard the instruction (this improves the accuracy of the pending tag count during and after the manage tag dialog)

and the same thing for 'petition tag' when the tag isn't already current

any accidentally added 'namespace:'-type tags will be deleted on update

fixed a bad merge in the manage tag siblings dialog's autocomplete dropdown lists

the thread watcher can now deal with urls with a #54951495 post anchor at the end

refactored some controller manager code

removed some useless old code

cleaned some misc code

improved some bad old orphan deletion code

deleted some old code

finished off some old media result streaming code

refactored a bunch of search stuff from clientdata to clientsearch

rewrote some subs gui/help text to be a bit clearer, and added a help link to the dialog

version 184

added external client_files storage!

you can add external client_files folders in options->file storage locations, further giving them weight

a new daemon will incrementally rebalance your files (and recover orphaned subfolders!) over your different storage locations

you can also force a full rebalance from the new database->maintenance->rebalance file storage

simplified the maintenance and processing panel

the maintenance and processing panel controls now appropriately support being set to 'none'

the 'run jobs on idle?' question is now explicit on the maintenance and processing panel

deselecting 'run jobs on idle/shutdown' will now disable subordinate controls

hitting tab on the autocomplete control now triggers an immediate 'fetch results' call

added a checkbox to options->speed and memory to completely disable automatic autocomplete results fetching (i.e. if you want to manually control a/c result-fetching only with this new tab shortcut)

the new less-laggy autocomplete results fetch won't trigger if the latest query is shorter than the cached query (i.e. you won't get lag when hitting backspace a bunch of times on autocomplete)

hitting shift+tab on the autocomplete control now disables all other key event capture, letting you enter IME without your up/down/enter presses controlling the dropdown list of results

'waiting politely' time is now indicated with a small 'traffic light'-type circle control on all downloader pages

fixed some occasional 'I'll go sit in the top-left corner of the screen and not fix my position' hover window behaviour in Linux

fixed sibling predicate collapse for the 'read' autocomplete dropdown for database results

refactored how predicates are collapsed and sorted to be a bit more sensible

fixed maintenance of local booru data use even if local booru does not receive any requests

improved some service/content update error handling in repository sync

the system:rating value-entry dialog no longer lists its rating services in random order

'open externally' will no longer show for non-local thumbnails' right-click menus

fixed audio/pdf thumbnail display for local booru, although they will be full size for now

harmonised some hex-prefix folder generation code

improved some file and directory copying code

improved upload pending popup message cleanup

added help_dir to hydrusconstants

miscellaneous refactoring

misc cleanup

extracted server-specific services and server resources from the common import path to the server import path

a bit more misc server refactoring

updated a couple of bits of help

version 183

added swf thumbnail support--it works ok for most swfs!

thumbs for existing swf files will generate on db update for both client and server

the server will also generate thumbnails for video on update, which it seems it was not doing before

rewrote some hash comparisons in thumbnail downloading and thumbnail counting code to be a lot faster and memory efficient

fixed thumbnail count invalidation after service sync

in certain cases, fetching autocomplete tag results from the db will be slightly less laggy

if the autocomplete dropdown has cached or otherwise quickly fetchable results, it will refilter for current input as you type, ignoring the normal character delay (i.e. typing through autocomplete is far less laggy now)

fixed a couple of autocomplete tag-parsing bugs for tags with more than one colon character

fixed some key event selection bugs in the autocomplete taglist when there are no results in the list

if there are no results in the autocomplete taglist, the typically caret-moving key events up/down/home/end will move the caret (rather than being uselessly passed on to the empty taglist)

all profiling now goes through a single location

profiling now also prints extensive information on explicit code callers

fixed db profile mode printing display for the new logger

added new pubsub_profile_mode (AKA log-killer)

all menu popup display and explicit memory cleanup is done through a single location

hover windows will now not show whenever a menu is open

hover windows will now not hide whenever a menu is open (useful when trying to go 'copy tag' off the hover taglist's menu!)

a keyerror bug when force_idle_mode was hit right at program start is fixed

unexpected serverside client disconnections are now caught and further request processing is cancelled

middle clicking on an 'empty' system predicate in the a/c dropdown now will throw up the dialog to enter a value and then pass that filled in system pred onto the new search page

duplicate parents (which can occur with overlapping grandparents) are now collapsed

the upload pending popup message cancels properly after 'server busy' events

whenever the client's services are iterated at the view-level, it will now be in random order, to stop sync bottlenecks choking other services (and anything else like this that might have been occuring)

fixed a bug with double-clicking a tag selection on the the tag censorship taglist

fixed the too-fast frame timings on some videos opencv had a problem parsing

loading many video types will now be just a little faster

remote files will no longer present the right-click menu option to share->copy->file

hitting a shortcut for manage ratings from the media viewer canvas will no longer work if you have no ratings services

the base scripts (client.pyw, server.py and test.py) now have a shebang line to facilitate non-Windows use

laid groundwork for external client_files locations

plenty of misc cleaning and refactoring

version 182

all printing to the log should now be unicode safe

some other, miscellaneous file-write locations should now be unicode safe

cleaned up some now-redundant unicode-bytestring conversion error handling

the animation scanbar will now draw a little darker while the animation is paused

the client now uses the same new log object as the server, so all logged data will be copied to the terminal, and all logged statements will be timestamped

the client and server's boot and exit statements are more harmonised

server backup is more log-verbose

server backup now makes a simple copy to 'server_backup' folder--no more _backup nonsense

all serverside requests will now print a line in the form 'PORT METHOD PATH HTTP_CODE in TIME TOOK' to the log

page up and down now work (again?) for the thumbnail view. adding shift also works for selecting a page at a time

improved some animation painting logic

improved some rating control painting logic

improved splash screen painting logic

improved all other, misc painting logic--many things should flicker less or render just a little quicker

ratings hover window will no longer re-layout (causing flicker) on ratings set

improved some media canvas painting logic to de-jaggify some zooming

in the manage options dialog, tag-related gui options are moved from the gui panel to the new tags panel

added 'apply all parents to all services' option to the tags panel

the delete key now removes active search predicates and tags in the manage tags dialog

fixed webm link parsing for rule34hentai.net

if you attempt to petition multiple tags, you will now be presented with a dialog asking you if you want to use the same reason for all the petitions

edit import folder dialog now has scrollbars and will resize itself based on your monitor size

if the static ffmpeg executable is absent from the bin folder, the client will now attempt to just call 'ffmpeg' in the normal system path

fixed some 'None media' data calculation bugs when media viewer closed during very fast slideshow

fixed a 'None media' mouse position bug when media viewer closed during very fast slideshow

all wx timers will explicitly stop on exceptions (which should reduce some types of error spam)

refactored client hydrus network session manager to a better position

wrote a better wma/wmv determining test using ffmpeg

refactored cv2 (OpenCV) out of the server's import tree

fixed some server boot crash error handling

cleaned some autocomplete matches-compiling code

deletepath and recyclepath will no longer throw an error if the path does not exist

server_messages folder is no longer referenced in the code--if you have it, feel free to delete it

memory cleans up a little faster after gui page deletion

version 181

fixed a potential bug in the server's db, very important you update this week if running on Windows

improved some thumbnail page calculation logic

improved some thumbnail page drawing logic

fixed broken vertical resize thumbnail grid issue

fixed some broken 'thumbnails have changed dimensions' event layout recalculation

fixed an idiotic typo bug that was making cached thumbnail page bmps taller or shorter than needed

thumbnail fading and other general thumbnail redraws should be a decent bit faster now

thumbnails have handed off all bmp storage responsibility to the thumbnail cache, which should mean greatly reduced memory use when browsing very large results

thumbnails will now only quick-draw if they have been seen before on the same page, which should reduced some page-refresh waterfall jankiness

thumbnail media panel double buffering seems to be working better

improved some general thumbnail drawing code

the media viewer background taglist will now show petitioned tags

the media viewer hover taglist should now look exactly like the background taglist (sorted lexicographically, no more counts, and now shows petitioned)

the 'selection tags' taglist will now show petitioned tags for 'all known tags' queries

the 'selection tags' taglist and the media viewer background and hover taglists will now count collapsed sibling tags correctly

all taglists now support ctrl+a to select all

improved some generic taglist value-setting code

parents should now only expand for their specific tag services

siblings, parents, and the pending count should now be correctly recalculated upon repository update processing

added 'copy sources' to the import status window's right-click menu so you can copy bad urls or whatever to the clipboard

added 'skip' to the import status window's right-click menu so you can skip urls you don't want

added default booru for rule34hentai.net

under certain circumstances, services with duplicate names could be created--any services like this will be renamed on update, and the loophole where this could happen is now closed

improved a locale number formatting call to ask for unicode

improved newgrounds parsing--mp4/wmv files should now work

the server logger has been formalised into a proper class

fixed some bad static image initialisation in the canvas code that was causing error spam

fixed default import tag options for hentai foundry and pixiv

copy->(bmp|path|local url) will now only appear as an option if the currently focussed media is local

fixed some gallery identifier print code

frames and dialogs will now clip their initial size to the current monitor's resolution--better scrollbar support for cramped windows will follow

updated ffmpeg static builds

updated sqlite command line exes

put off sqlite windows dll update because of critical server db bug

misc code cleanup and redaction

version 180

middle click on any taglist now does 'open a new search page for selection'

moved mouse idle checking code to the wx thread, which seems to have fixed a serious X11 crash

memory gunk should now be painted over when transparent animations are resized on linux

fixed some page up/down index calculation on the autocomplete dropdown

system predicates will remove themselves from the active predicate list silently again

system predicates will work again in queries that include tags!

greatly improved my string/unicode conversion code

dropped many inefficient unicode conversions from db code

improved the sql-grammar of some specifically efficient db code

removed incorrect unicode conversions from url/network request code

removed unicode conversion code from network response code

all directory iteration should now occur in unicode

all inputted paths should now be converted to unicode

replaced every single ugly raw path concatenation with a better method

improved some bitmap placeholder initialisation that was causing Linux terminal spam

added an option to always hide the preview window to options->gui

the preview window will not accept setmedia calls if it is hidden

if the preview window is actively hidden due to an unsplitting event, it will null its current media

fixed a shutdown calculation that was under-remembering the height of the preview window by four pixels every time

improved thumbnail waterfall code--it should be a little faster. less subject to lag when taxed, and waterfall with more genuine randomness

improved some thumbnail page drawing and clean/dirty conversion code

when pages are dirtied, any to-be-waterfalled thumbnails are cancelled, making superfast scrolling less crashtastic

new dirty thumbnail pages are now generated as needed, rather than precached

subscriptions will now correctly do a polite wait after fetching tags for redundant files

splash screen will now explicitly raise itself to the front on init

the export tag txts function of the export dialog now exports the tag txt to [filename].jpg.txt, rather than [filename].txt

the server now stores its updates in service specific subdirectories, like the client

some importers will better report when they are waiting politely

OS X (and anything else with OpenCV 3.0.0) now uses OpenCV to render gifs by default, falling back to PIL on error (like only Windows used to)

'open externally' from thumbnails now defocusses the preview window

refactored away a bunch of bad global controller references

added some shutdownexceptions

silenced shutdownexception terminal spam

reworded the 'maintenace and processing' options panel a little to explain things better

improved how the base_dir is figured out on program boot

improved serverside 'should I clean dirty updates?' check code

deleted some old code

split path-specific functions off from HydrusFileHandling to HydrusPaths and de-looped my import tree appropriately

general refactoring

version 179

all tag listboxes support multiple selection

all tag listboxes manipulate their data as sets of tags rather than individual tags

all tag listboxes report to their callbacks as sets of tags

many dialogs and other windows that use tag listboxes now deal with tags as sets

double clicking the media viewer's tag hover window now launches the manage tags dialog

the media viewer's tag hover window now includes (+1) counts like the manage tags dialog--harmonising the underlying canvas tag list will follow

lots of tag/sibling/parent code has been iterated over

ditched some misc redundant code and needlessly tightly coupled object relationships

some bizarro tag sibling dialog code has been rewritten

the autocomplete dropdown for 'writing' will now not expand search results to include parents when the receiving control isn't interested in them or it is otherwise not appropriate

when the autocomplete dropdown does have parents, they will be selected in a group with their child

the autocomplete dropdown for writing now broadcasts 'I'm done with tagging, close the dialog pls' in a better way

'activating' the sibling or parent dialogs' listcontrols (usually by hitting enter) now processes all rows that are selected, not just the first

manage siblings and parents dialogs can now take multiple initialising tags from the taglist right-click menu

removed some redundant listbox code

cleaned up and improved some tag list event processing

cleaned up some taglist right-click menu code

added system-wide mouse idle test to idle calculation. you can set this in files->options->maintenance and processing, and it defaults to ten minutes

import folders now support tag import options' explicit tags for multiple tags to any tag service

existing import folders will be updated to the new version, and if a local tag exists, it will be intserted into the new import tag options

fixed a hentai foundry page parsing bug

the deviant art downloader can now download >1024 pixel width versions of images via the download button (and a bit of cookie magic)

gallery page queries that 404 (like for a non-existent username) will now report 'Gallery 404' rather than spamming the gallery's whole custom 404 html page to the status box

fixed a bad layout flag that meant some namespace checkboxes in tag import options could remain hidden during first panel expand until first mouseover

the import status frame now initialises its status text properly

fixed last week's num_tags optimisations, which accidentally broke num_tags < x for x > 1

the client should work on both OpenCV 2.4.x and 3.0.0 (thankfully, the only difference for our purposes turned out to be some static variable renaming)

the Windows and OS X releases now come with OpenCV 3.0.0

increased the max period of import and export folders to 30 days

the launchfile/directory thread is now a daemon, so the (some flavours of Linux) client can shutdown even if an externally launched file/dir remains open

misc cleanup

version 178

import tag options now supports 'explicit tags', which will be added to all files imported

the default import tag options system supports explicit tags, and will propagate them to new import tag options objects

the gelbooru 503 Forbidden problem is fixed

all gallery downloaders now send the page_url as http Referer header when fetching their image_urls

pixiv parsing now grabs the correct tags for the image! previously, it was fetching the user's most common tags

for now pixiv manga (which is not yet supported), will be skipped over in the gallery parsing step

fixed a thumbnail drag-and-drop calculation that meant the DnD event would not start if the initial mouse movement went in an up-left direction

bad animated gif frame timing detection is improved

the fallback animated gif frame timing is scaled back to a more reasonable 12fps

fixed a critical bug in the manage ratings dialog that meant any NULL or MIXED numerical rating controls were mistakenly creating 'set rating to 0.0' content updates

fixed an obselete call with the new Pillow library that broke video thumbnail generation

added an explicit file import test to my test suite (previously, this was a manual test, and I did not notice the failed video files last week)

fixed a critical os x crash after navigating media with the manage tags dialog launched from the media viewer

improved speed and memory use of num_tags queries a little bit

reordered the predicate filters in the main query code so that num_tags happens right at the end, where it will run fastest

removed an unnecessary filter from the main query code

removed an accidental doubling of the simple predicate filters from the main query code where the file service domain is specified

improved an important db index and made sure the db uses it everywhere it can. it should speed up most non-'all known tags' queries

harmonised how all the new serialisable JSON objects are sent to and read from the db to a handful of simple general functions

externally opening a file is offloaded to a temporary thread to avoid gui lockups on OS flavours that have delayed 'open this file pls' calls

increased the commonly used read/write 64k block size to 256k, let's see if it reduces some file fragging

clientside hydrus bandwidth reporting is a bit faster and simpler

general code cleanup

general refactoring

a long-term refactoring job that aimed to better decouple the client, server, and hydrus parts is now completely done

delineated the layout of the gallery and page of images management panels a little

updated several links in the help and the program to https

the petition content checkbox list is now significantly taller

cleaned up the static folder a little bit

silenced an unhelpful webm last frame error

version 177

the old YAML "default advanced tag options" is converted entirely to the new JSON "default import tag options"

created a new JSON object to store new options objects

updated the gui to handle new default import tag options objects

existing default advanced tag options data will be converted to the new system on update

the petition management panel now presents similar petitions from the same user together, with the relevant data rows in a checkbox list, so janitors can approve/deny many similar petitions all at once

the petetion management page no longer auto-loads file results for a petition--you can now see specific results for each sub-petition by double-clicking on them

the server-to-client petition object is converted from yaml to json

the new petition object now stores all the different 'content' rows for the same service/account/reason/action

small content rows are now stored and transmitted in their own object

the network now supports POST JSON requests

account modifications now work through JSON

the account identifier object is converted from yaml to json

in the path tagging dialog, regexes that do not compile will now say so, reporting their short regex error statement

fixed a bug that could sometimes occur when cancelling an ongoing gallery query

cancelling an ongoing gallery query is now reflected in the gui much faster

repositories will not burn CPU cleaning their updates if there are still pending file/mapping petitions to process

repositories will clean updates in smaller bursts to reduce the contiguous server_busy time

fixed drag and drop thumbnail ordering, which actually wasn't working but was accidentally randomly sorted when I tested it!

a certain kind of error during vacuum that can mean two different things is now caught, and automatic vacuums then suspended while the user figures out what is going on

revised some v176 update code that wasn't parsing some downloader pages correctly--now, if a downloader page cannot be updated, it will be discarded

updated to Pillow 3.0.0 on all platforms and fixed some obselete image calls

general code cleanup

general code refactoring

version 176

the gallery downloader is updated to the new system

the gallery downloader code is generally a bit better all around

the gallery downloader will now remember its urls and file and tag options and paused status and so on throughout a reboot

the gallery downloader supports 'get tags even if file already in db' option

the gallery downloader management panel is updated for the new object. like the other panel updates, it is a little more minimalist

the gallery downloader management panel now has the import status button so you can review your parsed urls and retry them, or copy errors, or whatever

existing gallery downloader pages in gui sessions will semi-convert on the db update

subscriptions are updated to the new system

the subscription code is generally a bit better all around

subscriptions will no longer dump out on a failed file!

subscriptions can now recognise when files are uploaded during a page-walk (which shuffles everything up by one, of course), and recover without assuming it has synced

subscriptions can now be set an initial file limit and a periodic file limit separately

the manage subscriptions dialog is updated for the new object

the manage subscriptions dialog now has the import status button so you can review your parsed urls and retry them, or copy errors, or whatever

existing subscriptions will convert on the db update

subscription popup now shows per-file download progress

boorus that use number of thumbs to advance their gallery page indices can now resume

the thread watcher now obeys 'exclude deleted' status more reliably

the 'page of images' downloader now obeys 'exclude deleted' status more reliably

subscriptions now attempt to recover their sync status when a new file is uploaded during sync

fixed a bug where the thread watcher was not applying the filename namespace to new files, only redundant ones

the old advanced_import_options dictionaries are all updated to the new import_file_options object

improved subscription-dialog-was-opened-while-subs-were-running detection and reaction

the way galleries are identified behind the scenes is now much simpler and more flexible

for now, hentai foundry downloads and subs will no longer support the specific hf filter--they will be unfiltered and sorted newest first, so they work like all the other galleries. I may reintroduce 'sort by popularity' if there is demand for it

a bit of importer status update refactoring and cleanup

fixed some db vacuum maintenance timing that was interfering with gui reporting, so the 'currently vacuuming' popup never appeared until the vacuum was complete

fixed a possible bug for import folders that have a local tag set

splash screen is no longer stay_on_top--if there is demand, I will add an option

fixed a bug in the page deleted detection code

added page hidden detection code, replaced some ambiguous pause code with it

improved pause/play code throughout the program

corrected some bad x/y status texts

for simplicity, the import_file db command now always returns ( status, hash )

for simplicity, the import_file db command no longer takes service_keys_to_tags

plenty of general code cleanup

version 175

the 'regex shortcuts' button in the path tagging dialog now permits the managing of favourite regex phrases

added 'start animations this % in' options to options->media

animations will no longer clear to white while scanning beyond their immediate frame cache

modifier-click (like shift-click) will now drag animations without altering their pause/play status

the splash screen is a little larger, to better fit db update text in

the splash screen's text is split into two lines, and shutdown repository processing is a bit more verbose

fixed some incorrect dialog text in manage sibling and parent dialogs

rewrote sibling and parent pair update code to deal with unusual situations more gracefully

improved how the new downloader engine saves error notes

the import review frame now lets you copy the selected entries' notes to the clipboard

video files will no longer fail to import in windows because of unicode characters

recycling files with unicode characters on Linux will work more reliably, and when it breaks, it will fail more gracefully

the client and server can now detect and recover from a rare error where a database file is created but never initialised with the appropriate tables

the tag/file upload popup message now dismisses itself after a few seconds, rather than pointlessly lingering

wx event ids used in one-shot menus will be recycled, which should alleviate the 'after a long time, some menu selections stop doing anything' problem

some code cleanup

version 174

'server stop' and 'server restart' now work!

server will now log everything to console as well as server.log

server log entries will always be prefixed by the time they were made

server shutdown will close services more neatly

server shutdown cannot be triggered multiple times with multiple keyboard interrupts

server initialisation and shutdown is more verbose

server now accepts POST '/shutdown' requests from localhost

fixed a server shutdown bug

mousewheel events that hit the command and ratings hover windows will be propagated to the media viewer (e.g. if you click a rating, you can now scroll immediately to move to the next file without having to refocus the media viewer window)

fixed a bug where zooming out would go immediately to 1% if the media's exact fit zoom was already in the list of default zooms

added 'check db integrity' to database maintenance menu

multiple drag-and-dropped thumbnails will now be listed inside the drag-and-drop object in their hydrus thumbnail order (was previously random)

animations will now pause/play correctly during inbox filtering

most client errors now cause a 1 second wait on behalf of the caller, which should bottleneck error spam when it happens

fixed some buggy display and rescind behaviour with pre-dialog-existing petitions in the manage parents and siblings dialogs

the manage parents and siblings dialogs will now not attempt to remove/petition/rescind a pair if the entered list of pairs has entirely new entries

the manage parents and siblings dialogs will now attempt to maintain their sort order after you change some pairs

generally cleaned some parent and sibling dialog code

added option to options->files and trash for whether to use your OS's recycle bin

system:mime text now specifies specific mime group if appropriate, or otherwise lists every mime if >1 selected ('system:mime is specified' is dropped)

hydrus repositories will sometimes catch and recover from zero-length update files

hydrus repositories will catch processing parsing errors and pause when they happen

the new hdd, page of images, and thread watcher import daemons will tidy themselves up if their accompanying pages are completely deleted

improved how daemons shut down

improved client shutdown logic and speed

some misc code cleaning

more general code cleanup

version 173

converted import folders to the new system

import folders now have names, so you can have several pointing to the same location

import folders can filter by mime

import folders now have separate actions for successful, redundant, deleted, and failed files

these actions can be 'leave alone', 'delete' or 'move to location'

import folders now support import file options

import folders can now be individually paused

import folders' popups are individually optional

import folder edit dialog provides a path cache button, like new download pages, to let you review the paths it is remembering

existing import folders will be roughly converted to the new system on update, but they will be in a paused state, so you can check they are what you want

created new mime selection control

mime selection control added to mime system predicate selection dialog, so you can now search by mime more specifically

updated some mime system pred code to deal with this change

a single non-dragging left-click on an animation now pause/plays!

the animation scanbar will now only resume playing the video/flash after a scanning click/drag if it was playing before the drag began.

added a 'polite wait' option to options->downloading. this is temporary until I write a better clientside bandwidth throttling system

added a 'remove/petition all tags' button to the manage tags dialog to make mistakes easier to clean up

added 'confirm send to trash' to options->gui

added 'confirm archive/inbox' to options->gui

added a simple 'export tags in .txts?' button to the export dialog. In future, I will expand this to do service-specific exports and also add it to regular export folders

fixed some invalid name error handling in the manage dialogs

fixed a bug that sometimes occured at the end of an admin immediate sync

fixed a rare shutdown-order error in video rendering

fixed an animation bar initialisation error that was causing error spam on some videos/flash

general import code cleanup

cleared up admin-side petition wording and presentation

improved the threaded workflow and error reporting of the admin-side petition page

the thread watcher now reports its number of next seconds until the next check in properly formatted time

version 172

the server no longer needs wx!

the server now runs from command line

the server is now .py, not .pyw!

run 'server help' for more info

improved how the controllers start

improved how the controllers stop

improved controller boot error handling

improved how pages close

general controller code cleanup

general server code cleanup

fixed the always_on_shutdown (without asking) option, which was asking anyway

removed a debug statement in isalreadyrunning code, whoops!

improved isalreadyrunning detection. it should work breddy gud now

added gallery file limit to new 'downloading' options page, which folds in the old thread checker options as well

added option to always embed the autocomplete dropdown window (rather than having it a floating window), which is now default on for Linux and OS X

manage services now supports two kinds of service reset for repositories

the service reset buttons now only fire on dialog ok

service reset will try, as cpu allows, to update its progress in a message popup window

administrators now have a 'sync now' button on the review services window that lets you catch up immediately to the service without having to wait for the normal update time (this will burn cpu time serverside, so be careful!)

fixed a bug when searching boorus with unicode-16 characters

the client updates directory is neater

'system busy' status is now shown on the status bar

'force unbusy' added to debug menu

invalid characters in export filenames will now be replaced by underscores

fixed a bug where rating services' cached file counts were not decrementing on de-ratings

rating services' cached file counts are reset on update

hover windows will pop in over video again, but will not if the mouse is _near_ the animation bar

searching for ':' in the autocomplete dropdown will no longer search the db for every single tag jej

improved some server_busy logic

improved some server shutdown logic

version 171

improved isalreadyrunning code

fixed isalreadyrunning code for the Linux frozen build

the client's isalreadyrunning splash check will respond more quickly if you choose to wait

repository processing sync will no longer update the frontend, which approx doubles its speed

repository processing sync now only occurs when the client is idle, so it won't slow your browsing down

if you catch the client while it is processing, doing pretty much any action will make it quit out nice and quick

repository processing sync now occurs at a lower db level, meaning less laggy popup ui update and overall faster processing time

repository popup disappears after a few seconds once it is done--if you leave your client on all the time, you probably won't see it again.

split 'maintenance and memory' options panel into 'maintenance and processing' and 'speed and memory'

added 'cpu busy' option to options->maintenance and processing

added 'run stuff on shutdown' option to options->maintenance and processing to set whether pending db maintenance and repo processing can happen on shutdown

added 'max minutes on shutdown' option to options->maintenance and processing to limit how much shutdown processing can be done in one sitting

repositories can now sync on shutdown--they will report through the splash window

shutdown maintenance is checked and run in a better way

moved to a two-stage shutdown procedure

daemons are better kept track of, and they talk to the controller in a better way

daemons shut down in a better way

improved a bunch of idle/busy/just woke from sleep logic

improved how the controller maintains the db and memory

improved force_idle mode in the client

idle calculation is more accurate, and resets on more user-driven events

improved responsivity for processing popup

fixed a serious bug that was counting some unprocessed content as done when the service processing window was cancelled

misc code improvements

misc maintenance code improvements

fixed a bug with parentless center-positioned dialogs not sizing themselves properly

removed delete_orphans timespan param option

fixed bugs selecting and media-viewering collected media

fixed a missing thumbnail selection bug

fixed some os x dialog interaction and z-order problems when the splash screen is open

os x splash screen is no longer stay_on_top

fixed the 110.00% display bug

removed wx from pubsub code

pubsubs now route through the controller

reduced some pubsub overhead, we'll see if it smooths out some of the gritty stuff

misc thread worker pool is now managed by and routed through controller

some other thread-wx interaction is now routed through clientcontroller

db now interacts with controller directly, not through wx

all other indirect references to controller go through hydrusglobals, not wx

refactored everything wx away from everything hydrus/server except controller code

about window now says the client's network version

version 170

added media viewer mime options to file->options->media page

you can now set animated mimes to start in a paused state

you can now set the client to 'open externally' any media from a thumbnail activation

reworked the way the media viewer generalises and displays its media to obey these new options

improved some misc zoom code

flv video is now rendered natively by default

audio now defaults to an 'open externally' button

renamed the nebulous 'url' download page option to 'page of images'

moved the 'page of images' download page to the new system--it'll now remember its state through a restart, has a the detailed import status button, all that stuff

'page of images' download page now supports checkboxes to customise which types of links it will search for

you can now close and rename page tabs from a right-click menu!

deviant art parser now works for mature images

improved deviant art tag and image page parsing

added 'remove filtered files' option to file->options->files and trash

improved UPnP dialog error reporting for when external IP cannot be parsed and for when external IP is reported as 0.0.0.0

added an external ip/host override option to file->options->connection page

whenever the client or server deletes a regular file or directory, it will now send it to the recycle bin

files stored in the db are no longer read-only

existing files stored in the db will attempt to be set read&write on update

wrapped undo manager access in a lock, which may stop index bugs when a lot of stuff is going on

fixed a bug in download progress->progress gauge hook when content length was absent

removed previously entered zero-length namespaced tags (like 'wallpaper:') from client and server

moved 'client/server already running' check from db access test to more reliable process list review

improved 'client is already running' mini-dialog

added cpu percent utilisation check (if any cores >50%, dump out) to idle check

if the 'canvas zoom' value exactly matches a default zoom, it will not be inserted into the list of zooms to scroll through

the 'canvas zoom' will be displayed with hundreds of a percent accuracy

the new thread and hdd import pages will now obey page-global pause/resume events

the new thread and hdd import pages now have the 'sort by' dropdown

the new thread import page supports set_search_focus event

added import tag options info to 'getting started - tags' help page

relabelled the path tagging button on the import files dialog to something that made more sense

might have patched an unusual subscription error reporting bug

updated sqlite on windows--db should be a bit faster

updated pillow on all platforms

compiled a new python for my os x build with newer openssl library, which should fix some ssl problems people were having

updated some deprecated directory parsing code

general code cleanup

version 169

revised all the help files

updated out of date screenshots in the help files

added some more screenshots to the help's index

cleaned out unused help images

fixed some missing service error catching in the custom filter setup dialog

fixed random reordering of paths from import files selection dialog

logged errors should now be prefixed by the time they occured

errors should now be more reliably printed to the log

fixed a problem with e621 queries that included '/'

improved some server temp file error handling

fixed a typo that was throwing errors when trash settings were set to 'no limit'

version 168

thread watcher is moved to the new system--it will remember its previous state

thread watcher has more compact, flexible gui layout

thread watcher now supports file import status button, so failures can be reattempted

thread watcher check now button has improved logic

import tag options and import file options gui controls now plug into the new system

thread watcher will now remember its tag and file options

thread watcher will accept changes to its tag and file options after it is started

fixed a bug in import tag options deserialisation

hdd and thread watcher static texts will flicker less

improved some network transfer gui reporting code

send to trash, delete from trash, and undelete now all have different icons in the command hover window

added open externally button to command hover window

opening an animated gif or movie externally from the media viewer will pause it in the client

share->copy->image added to media viewer for static images

fixed a typo bug in copy 'image' to clipboard

fixed a typo in external ip discovery that was causing errors for local booru external link generation

improved external ip discovery error recovery

in prep for the de-wxing of the server, refactored a lot of code so the server and client code don't import each other at any point

if an attempt to close the client is aborted (e.g. because of an active import page), the exit splash screen will now destroy itself

version 167

created gui control to show file import status

added a button to launch this control, wrapped in its own frame, to the hdd import page

you can now drag and drop thumbnails out of the client for quick export!

the client will catch internal drag and drops, and won't interpret that as an import request

dragging files onto the client will now always show the 'copy' mouse icon

some thumbnail generation code is a bit quicker--resized thumbs are now always generated on their first viewing

if you use 200x200 thumbnails, the client won't bother to generate resized thumbs, saving space and time

the text entry dialog will now check its button status on a non-keyboard related text-change (e.g. selecting paste from the right-click menu)

the popup created by downloading a raw url or a youtube video will finish itself properly when the import attempt is completed, removing the pause and cancel buttons and making the popup dismissable

the splash screen will now stay on top

the password entry dialog will also stay on top

you can now copy the md5, sha1, or sha512 hashes of local files from the normal share->copy->hash menu

fixed an overly broad deleted/redundant test on md5/url pre-import status checks

options to remember size of manage tags and position of both manage tags and ratings dialogs added to file->options->gui

updated some of the help

version 166

created new object to hold hdd import information

created new object to hold generalised import file status

moved hdd management controller to the new system

moved hdd management gui to the new system

hdd imports will now remember their import and pause status through a session change

misc import code improvements

hydrus client is getting out of the zip business--zips are no longer parsed for import nor able to be created for export

import and export code is simpler

repository file downloading daemon will now throw up a small auto-dismissing popup when it downloads files

repository file downloading daemon will respond to new downloads much quicker than previously

if a repsitory is in the process of backing up when it receives a request, it will now return a 503 'server temporarily busy' error (rather than timing out)

the client can now catch and handle these 503 errors gracefully

backing up will no longer block the client's gui, and it will no longer timeout if the operation takes more than ten minutes

some networking code is a little simpler

added 'select local files' and 'select trash' to thumbnail right click menu

'cancel download' added to file repository submenu of thumbnail right click menu

some buggy listbox resize behaviour (scrollbars not disappearing and occasional layout and drawing update fail) has been fixed

version 165

added a db table to track when a file was sent to trash

added an option for whether trashed files are removed from view

added a max age to trashed files

added a max size to the trash

added a daemon to maintain the trash

improved some generic deleted shortcut logic, so delete key will present option to delete from trash if appropriate

shift+delete will now undelete

misc trash code improvements

thumbnails deleted from the trash will accurately remove their inbox status icons

images downloaded from file repositories will accurately add inbox status icons

reduced chance of problems when trying to delete media from the trash when it is currently being rendered

further reduced this chance

removed redundant undelete code, folded it into add/delete code

the media viewer will now remove files deleted from the trash in all cases, even when launched from a file repository search

significantly improved how animations figure out when to put frames on screen. these timings should be much more accurate, slowing down only if your CPU can't keep up

8chan thread watcher now parses all files from multiple file posts

if a booru provides a link starting with 'Download PNG', which refers to the original source png, that will be preferred over the jpg (konachan and yande.re, which run Moebooru, do this)

booru parser is now a little more liberal in parsing for the image link

yande.re booru support is added by default

fixed some local/remote state code, which was breaking file repository searches in several ways

improved error handling in repository file download daemon

cleaned up manage options dialog code

reduced min size of the media caches in manage options dialog

moved thumbnail size to 'maintenance and memory'

added better error handling to repositories that cannot find an update file during processing

repositories that have such errors will pause to give the user a chance to figure out a solution

misc code improvements

fixed a bug where file repository downloads were not importing if the file had once been deleted

dropped the clever indices from last week--sqlite had too much trouble with them. I will reform my db tables next week to get around the whole issue. for now, I have reintroduced the bulky index and forced sqlite to use it where appropriate

added a test for collecting pending tag data

tags in the form 'text:', which would sometimes slip through when typing quickly, are now not processed

improved tag cleaning error reporting

improved when special wildcard and namespace predicates are provided

namespace predicates now display as 'namespace:*anything*'

fixed a bug when launching the media viewer from collected results

fixed a command hover window layout bug that was putting namespace text in incorrect places

fixed a bug that was causing the new client-wide processing phase option not to work because, ironically, its calculation was out of phase

review services will now state how many updates have been processed

review services will specify if an update is imminently due, rather than saying the repo is fully synched

fixed a review services layout bug that was misaligning text after an account or data usage update

fixed a bug in system:similar_to serialisation, so pages with this predicate can now be saved to a session

fixed the same bug for system:hash

vacuum should be a bit politer about waiting for its popup message to appear

database maintenance won't run while a repository is synchronising

version 164

rewrote the drawing code for the listbox that displays tags in various ways to be a lot faster and more memory efficient

updated one new client mapping index that wasn't working quite as I wanted it to something more clever

db will be a little smaller and mappings stuff will be even faster

merged the two ratings system predicate input panels, so you can now select like/dislike and numerical ratings system predicates at the same time

fixed booru download page serialisation, which means they will save to sessions

prototyped trash service

locally deleted files will now be sent to trash

locally deleted files will not be removed from the existing search

files can be permanently deleted from trash, which will also immediately physically delete them from your hdd

files can be restored from trash back to the local file service

inbox state is now more separate from the local file service, so it will be remembered through a visit to the trash

improved delete code all around

general inbox/archive db code improvements

misc content update pipeline improvements

optimised mass-adding of files to a service (for instance, when (un)deleting a whole bunch of files!)

delete orphans daemon is removed--it will be replaced by a more thorough single-shot hdd/db purge like 'check file integrity'

files are not yet automatically removed from the trash--I will add that next week.

updated db access info in db folder

added sqlite command line executable to db folder for all platforms

bit of code cleaning

cleaned up some gui error reporting

might have fixed a service cache bug in the db that was causing double bandwidth reports and possible looping sync behaviour

version 163

reconfigured some important mapping indices in the client db to reduce search time for many common tag operations

the new indices have also sped up tag processing significantly

added an automatic db optimiser analyzer run whenever the db updates

pixiv now downloads the largest version of an image again

pixiv tag parsing improved

added support for some unusual mp4 types

the noquery media panel is removed. pages with no query status will now show a normal thumbnail panel with 0 files, rather than 'None' files.

check file integrity will now report number of missing files and incorrect files separately

check file integrity can now take a folder to move incorrect files, rather than deleting them

when you try to pend a tag with a sibling in the manage tags dialog, you will be prompted with the chance to pend the sibling instead

the rating system predicate dialog now allows for easier (and multiple!) selection of like rating parameters

the rating system predicate dialog now allows for easier (and multiple!) selection of numerical rating parameters

the rating system predicate dialog now offers correct predicates for 'zero allowed' numerical rating services

some bmp icon code cleanup

changed pause and cancel buttons to bitmap buttons

cleaned up a lot of pause and cancel code

now, when popups are paused, they will hide their details

popups that are cancelled by a dialog change (for instance when subscriptions are changed while subs are processing) will now dismiss themselves after a few seconds

during repository sync, http failures are recovered from more gracefully, and any pending processing will continue

the repository update downloading and processing loops will wait on the db and gui more efficiently

fixed a bug in the way the manage export folders dialog was testing existence of subdirectories

general code cleanup

bit more code cleanup

some good daemon refactoring

version 162

on update, all previous gui sessions are deleted!

on update, all export folders are deleted!

made an important efficiency improvement to the new serialisation protocol--network version is incremented as a result

finished extracting management data from management panel

management data is wrapped in management controller, a new JSONable class

pages collapsed to a single class

page and management panel instantiation completely rewritten for the new system

created new JSONable guisession class for better session management

db is updated to store all this stuff in JSON rather than YAML

session loading is much less bloaty for clients with many sessions

all page types are now serialisable and hence addable to a session

moved predicates to JSON

moved filesearchcontext to JSON

autocomplete dropdowns now remember their tag service, include current, include pending and synchronised status over sessions

export folders now remember their file and tag services, and also include current and pending status

some general improvements to export folder code

cleanup of export folder dialog code

added a test for export folders

some general refactoring

some general cleaning

significant refactoring of predicate variables

fixed a bug in multi-version update from before v154

if you do not have any ratings services, the 'manage->tags/ratings' submenu will now just be 'manage tags'

changed some thumbnail menu entries to say 'selected files' instead of 'all files', which is more accurate, see if you like it

version 161

updated windows python, sqlite and ffmpeg

added linux ffmpeg binary to the executable build

updated os x ffmpeg binary

animation is generally working for linux and os x

fixed some linux/os x ffmpeg calls

fixed webm import for linux/os x

webms work great

moved to rendering gifs with PIL by default on linux and os x, which has bad palette support but renders more reliably

the animation scanbar position caret is now visible on linux and os x

fixed some timing/resume issues the animation scanbar on os x

the old mediactrl video embed is removed

mp4, wmv, mkv are added to native rendering control, no audio yet (just like webm)

flv and flash will now have a little vertical padding on max zoom to make vertical hover windows easier to access

changed database to a faster synchronisation mode

tuned update processing daemon for more accurate time calculations and faster baseline speed, even if that knocks gui latency a bit

fixed rows/s being thrown wildly off by long pauses

made update processing daemon more reactive

improved some of the flow and generally cleaned the update processing daemon

added processing phase option to regular delay to processing of updates (this is useful if you run multiple clients and don't want them to process at the same time)

fixed e621 tag parsing

completely rewrote the focus engine behind the autocomplete dropdown

fixed a bunch of autocomplete dropdown's generally buggy behaviour for linux and os x

fixed autocomplete dropdown's display, sizing, and positioning for linux and os x

fixed autocomplete dropdown selecting system predicates with the keyboard

fixed autocomplete dropdown show status in linux with multiple pages open

improved all subprocess calls, removing interim shell step and parameterising passed arguments

improved how files and directories are launched in windows

upnp is fixed for linux

fixed hover window archive/inbox/delete icon buttons, which were spamming their commands to all open media viewers, not just their own

added a simple raised border to hover windows to better delineate them from the canvas background

updated the hover window size and position code to be a bit more reliable (still seems to bug out a bit on linux)

tags hover window should now expand sideways more reliably

it is now not possible to create nested export folders

searching for numtags < x will now include files with no tags

fixed manage boorus dialog OK in linux

fixed booru selection mini-dialog in linux

generally improved restoring database code

fixed restoring a database when you have tag archives in existing database

fixed a unicode encoding error when converting certain jobs to text (this was throwing errors in deviant art downloads/subscriptions--we'll see what was actually going on now, whether this is an error not being formatted right or something else)

started some great rewrite of management panel gui code

rearranged the download panel gui hierarchy slightly

improved collapsible panel collapse/expand layout code

removed code and database table for the old numerical ratings filter

some general code refactoring and cleanup

improved client upnp daemon timing

harmonised how the client and server check requests for bandwidth-tracking eligibility

version 160

added options for http, socks4, socks5 proxying

improved some network-related errors

ratings services can have custom border and fill colours for their various states

ratings services can now also be squares or stars instead of circles

numerical ratings services can disallow zero ratings

fixed the JSON parsing error that broke the thread checker

shrank the width of the thread checker to make it a little less ugly

the autocomplete tag search dropdowns will now accept and search with quickly entered text

this new system will substitute siblings in the manage tags dialog

fixed rows/s average calculation

rows/s is more accurate

content update popup string update is less laggy

content update popup now shows content rows, rather than content parts

removed update 'taking a break' component, as it was not doing the job I wanted it to

db debug profile mode can now be turned off lol

fixed an error from middle-clicking greyspace in the linux notebook tab area

general code cleanup

some string conversion code cleanup

fixed a missing canvas bmp error with flash/flv embed buttons

version 159

split previously monolithic repository updates into smaller pieces

added service_update server calls

extended content_update server calls to support sub-indices

sped up some content update preprocessing code

improved some content update preprocessing code

radically reduced serverside memory usage while generating updates

added iterator splitters to make sure any single update row cannot be too large

thanks to iterator splitter, updates should process through the client more smoothly

added timespan splitter to make sure any single server update query cannot be too large

content updates are resumable if broken part way through downloading the list of them

the update popup will state how fast it is currently working in rows per second committed

cleanup of a lot of update related code

more cleanup of update related code

improved serialisable protocol so it'll work better over a network

made serialisable protocol much simpler

fixed numerical rating system predicate dialog slider range

fixed numerical rating system predicate dialog OK for valued predicates

fixed numerical rating custom filter action dialog

improved some network yaml error handling

replaced 'export tags' thumbnail menu entry with a tag archives system

replaced 'import metadata' file menu entry with a tag archives system

disabled thread dumper and manage imageboards and manage 4chan pass dialogs for now, because dumping code is out of date and completely screwed up

client will now not start in idle mode

help debug menu has new 'force idle mode' entry

idle mode is displayed on status bar

simplified client main gui status bar display and code

the flash and flv files' embed button now has a little border to delineate it from the canvas background

fixed some clientside bandwidth tracking code

removed some old networking code

made some custom objects draw themselves more efficiently

for now, manage tag parents and siblings dialogs will not show deleted rows. I will eventually add a 'show deleted' checkbox, like the manage tags dialog has

some static image rendering is slightly less laggy, particularly when browsing large images

fixed initial height of manage tags dialog launched from preview window

changed the 'search area' vs 'preview' sash gravity so the preview area won't expand or shrink on resize--see if you like it

autocomplete dropdown should now hide itself when focus goes to other programs

autocomplete dropdown should now hide itself when focus goes to other hydrus frames in os x

fleshed out new URLCache object to handle better gallery download url management

refactored some file status tracking stuff to a better system

refactored gallery page fetching to a better system

refactored gallery url handling to a better system

some redundant import checking should be much faster

improved error handling when booru image page parsing fails to find image url

refactored how tags are fetched for DA, tumblr, and giphy as part of the above overhaul

misc code cleanup

version 158

subscriptions now have a 'get tags even if file already in db' checkbox, defaulting to false, which will significantly speed up subs with redundant files

subscriptions will now more accurately obey initial limit

gallery downloaders will now more accurately obey file limit

merged some hentai foundry code into the general downloader

cleaned and refactored a lot of import page initialisation code

cleaned and refactored of lot of import management panel code

cleaned and refactored of lot of import code

created numerical rating control

added numerical rating control to rating dialog

added numerical rating control to rating hover window

updated like and numerical rating service info for future support of custom colours and shapes

fixed listbooks page deletion display

fixed manage boorus saving changes to existing panels

fixed manage imageboards saving changes to existing panels

fixed manage imageboards saving remove imageboard

improved and refactored listbook code, further separating active and proto pages

improved some listbook name conflict error handling

misc cleaning

export folders and the export dialog will also export file attributes like access and modification time

export folders will now overwrite destination files if they are a different size to the exportee

export folders can now be set to 'synchronise', and so will delete any other files in the directory. existing export folders will behave exactly as before

improved misc export and import folder code

moved folder type tooltip to an explicit bit of text on export and import folder edit dialogs

thumbnails can now display multiple hydrus file repository icons (current, pending, and petitioned) if multiple states apply

hydrus file repository icons are gone from media viewer--now there is a text list

hydrus file repository text list is added to hover window as well

exclude_deleted option will no longer hide files from file repository search results--it was more confusing than helpful

improved some error-prone logic in how advanced_import_options were being stored

ratings hover window icon background colour is fixed

imported bmps should now be converted to pngs with 1:1 colour (before, they were being collapsed to a 256-colour palette, which was showing ugly dither on complicated images)

fixed a sometimes segfault crash on search refresh on OS X

the preview window now supports a simple right-click menu

added 'open installation folder' to file menu

improved some media viewer precaching code, which should speed up some scrolling, particularly on the first scroll

improved some layout code

improved some file permissions code

cleaned some canvas code

version 157

fixed a bug in listbook page initialisation and retrieval that was affecting many dialogs on OK

some general dialog OK code cleanup

fixed a media-navigation bug in managetags dialog

fixed a serious OK bug in imageboards dialog

created a new 'periodic' object to manage future subscriptions timing improvements

started subscription YAML->JSON conversion

stopped compressing json in the client db so it is human readable and thus easily editable

subscriptions are no longer renamable, as this code was super buggy and could delete other subs

tidied up the database menu

a bit of misc cleanup

in many cases where it was once pseudorandom, services are now reported in alphabetical order

prototyped 'like' ratings control

added new like ratings control to the background bmp of the media viewer

added new like ratings control to the manage ratings dialog

added new like ratings control to a new hover window in the top-right

added basic additional icon support to new hover window

fixed some misc new alignment bugs related to new ratings stuff

like ratings controls on the hover window have tooltips

fixed up some icon/rating display logic in the background bmp of the media viewer

updated ratings dialog error handling

version 156

improved my build workflow in several ways, which should result in fewer orphan files and other build weirdnesses

some bad path usage in initialisation icon resizing has been moved to better temp paths

hitting page up or down on a manage tags dialog launched from the thumbnail grid no longer clears the current media

improved file permissions code across the program

fixed import folders daemon's test code for non-windows

fixed up some temporary file code that wasn't cleaning up those files when the application was about to close

fixed a newline parsing problem in copy/paste tags in the manage tags dialog

added tag cleaning to pasting in the manage tags dialog

added newline removal into standard tag cleaning process

fixed a server db bug that was stopping some accounts from being created

fixed some network session exception creation and catching

new popup messages should no longer steal focus in most circumstances

client should recover from serious popup message manager errors better

hover windows will now only pop up if their media viewer is the currently focused frame

hover windows will not hide until the mouse moves off them when flash or webm are underneath them

os x will no longer vanish media in the media viewer on an action like archive or inbox

fixed juddery media mouse dragging in linux

improved the way listbooks work to avoid a problem with clientdata in wx linux

export folder is gone as the default export location--now it is 'hydrus_export' under the current user's home dir

updated windows ffmpeg to latest version

fixed an important opencv dll conflict that was causing some gifs to render wrong in windows

shift focus media logic improved--shift initial thumbnail is now last image selected

shift selection will no longer deselect anything

version 155

fixed a frame seek error when looping long and/or large gifs with unusual palettes

improved recovery when fps reads as 0 on videos

added error detection and graceful recovery and reporting for missing ActiveX flash control

removed a lot of old imported messaging code that was slowing down boot

removed some other old library imports that are no longer needed

gracefully silenced 'application shutdown' daemon errors (they were previously spamming to console)

fixed a hover window display check that was sometimes spamming linux with edge-case errors

slideshows will now pause while the right-click menu is open, stopping bizarre full program crashes occuring on many menu items while slideshow was in progress

fixed a thread_id-sqlite-cache_initialisation problem that was breaking tag archive sync

harmonised copy/paste tag protocol in manage tags dialog with 'copy all tags' of the tags list

removed some old messaging code in client db creation

version 154a

fixed a bug in v154 update code when there was more than one set of shortcuts to convert

fixed a faulty default value for num_pixels system predicate that was stopping options from opening

fixed an error when video fps is 0

version 154

managed to completely break my linux and os x dev environments trying to update python--their release will come soon, once I've cleaned them up

removed some old cv code

updated to opencv 2.4.11 on windows

fixed a bug in trying to upload a small number of tag petitions

hover command buttons are now shrunk to exact fit size

added zoom buttons to command button hover window

added navigation buttons to command button hover window

added fullscreen switch button to command button hover window

added archive/inbox button to command button hover window

added delete button to command button hover window

added generic close window button to command button hover window

added tooltips to all the command buttons

cleaned some misc canvas code

hover windows now fit better, without overlap

moved media info strings to top of media canvas, so they hide behind hover header

hover windows will now not cover webms or gifs when the mouse is over the media container

hover command buttons now work for inbox filter, including back/skip buttons

inbox/archive button feels and works correctly for the inbox filter

updated manage options dialog's 'default file system predicates' mess to use the new cleaner predicate panels

all file system predicate defaults have been reset to default

added system:num_pixels to file system predicates default panel

fixed the system:num_pixels predicate being broken on options save

removed the ratings system predicates from file system predicates default panel

fixed misc system predicate bugs

moved yaml->json conversion forward:

settled on object code

added built-in compression to serialisation

added db tables and access code to support it

fixed a little layout mess in the edit custom filter action dialog

created a rich 'shortcuts' class for storing shortcut->action information that can be easily expanded to handle mouse events as well

new shortcuts class works on new json storage rather than yaml

moved custom filter shortcut action storage and general handling to new shortcuts object

moved old favourite_shortcuts tests to new code

cleaned a lot of the custom filter dialog code

rewrote the custom filter setup dialog to work more like other dialogs--now all changes are saved on ok, and save/save as/delete is replaced with add/delete

auto-creation of 'previous' shortcuts is removed, as all changes are saved to shortcut sets anyway

removed the ratings filters--the numerical filter may make a return in another form, your thoughts would be appreciated

fixed a cache counting bug when archiving redundant files during import

sped up file deletion a little bit

added an inbox cache to reduce laggy inbox checking, particularly search result fetching on dbs with large inboxes

fixed subscriptions that have no initial file limit, please reset your sub url caches to fix in these cases

did a lot of server/client database merging and refactoring

improved some database error handling

fixed an error in file repository superbans

optimised some critical db code

sped up tag censorship filtering

added a db profiling mode, accessed from the help menu, that will dump copious db profiling info to the log

misc code cleaning

improved some popup mesage print sync timing

rewrote some server auto-setup code to deal with slowly-starting server

version 153

cut out some out of date stuff from help

put a nice big red warning at the top of the 'running your own server' page

added system:num_pixels for megapixel searching

system predicates height, width, ratio, and the new num_pixels are now collapsed to a super-predicate, system:dimensions

system:rating will now only show if you have some ratings services

the system:rating dialog will now hide like or numerical ratings if you have no services of that kind

complete refactorisation of system predicate dialog

some misc system predicate bugs and bad panel event precedence fixed

shortened all predicate spinctrls

improved how system predicates are stored and used for a search

moved some view menu items to the download menu and the new search menu

all of the download gallery pages are added to the download menu

added some explanatory text to the pixiv dialog

the tag uploading process will more carefully prepare its subupdates so that mass allocations of the same tag to many files will be split into smaller chunks, avoiding connection timeouts in these cases

hover windows will no longer hover over flash files

tag hover frame will resize a little based on the canvas width

the twenty pixel mouse warping when the mouse starts dragging close to the edge of the media viewer will no longer occur on flash files

the canvas buffer for flash files is expanded to five pixels either side

the hover tags window will now update itself when the media's tags change

new hover window for details and commands, up top of the media viewer

moved the old popout windows' buttons to the new hover window

added hover window commands for the normal browser

cleaned up some general canvas code

index string will now display with a slash in all cases (it was a backslash for windows, for accidental reasons)

file limit for subscriptions and download pages now has minimum value 1

searching for number of tags while tag censorship has some namespaces  will no longer throw an error

moved boot and exit code responsibility from the splash frame to the controller

improved the boot and exit code generally, including dragging feedback lag and error handling

improved some thread error handling

fattening service info won't trigger on client shutdown quite so much, which should speed up shutdown a little

version 152

added prototype hover frame for tags in media view

hover frame shouldn't show when a dialog is open

manage tags dialog launched from media viewer now has a delete button

subscriptions now have an 'initial limit' variable, defaulting to 500, that limits the total number of files the subscription daemon will look for on the initial sync

added a similar file limit spinctrl to gallery download pages

updated layout of import files dialog

cleaned up some ugly global variable scope stuff

fixed initialisation of advanced import options in file import dialog

made a good start to better object serialisation

subscription and repository sync daemon jobs that stop due to a dialog-driven change during processing will now cancel themselves after five seconds

fixed namespace (e.g. 'series:') tag censorship

fixed the fullscreen switch bug that was breaking an (initially fullscreen)->(non fullscreen) media viewer

fixed some search logic (some system predicates were not firing when there were no regular tags present)

removed some artificial delays on daemon db access, let's see if it chokes anything

harmonised a bunch of client and server controller code

created a common controller class and merged a lot of the client and server controller code into it

general code cleanup

more general code cleanup

version 151

added a possible solution to the manage tags dialog next/previous buttons crash

completely overhauled temporary file management throughout client and server

removed old temp folder

copy files to clipboard will now copy the database's exact file paths (rather than copying to a temp folder beforehand), making it a lot quicker

the import folders daemon will no longer attempt to import (and hence spam errors about) zero-length files

added one-time tag archive sync button to the 'perform a service-wide operation' dialog in review services. this lets you add or delete the archive's tags by namespace

revised the tag archive sync code a little so it syncs to non-local files as well, if the hashes can be lined up

added hash cross-referencing to archive sync code before maina processing step, often saving a huge amount of time

fixed a parsing error with advanced content updates at the gui-end

added GetName and IterateHashes to hydrus tag archives

added default 'similar to' max hamming distance to file system predicate defaults in client options

fixed up the default text display of the file service system predicate

added some text to the sort/collect options panel to better explain what the big listbox is for

fixed a popup numerical typo when uploading files to a repository

fixed an utterly bizarre bug caused by the import files dialog that was screwing up subsequent dialogs' layout and size, apparently by wx voodoo

refactored all the functions and classes of hydrusconstants to better locations

misc other refactorisation

misc cleaning

renamed hydrusdownloading to clientdownloading

changed all default { } and [ ] parameters to None. I don't think it mattered, but just in case for the future

moved some ugly global variable stuff to better places, hoping to do more in future

cleaned up many unused imports

version 150

added an 'idle' updating processing mode that churns through tags four times faster but makes gui laggy. it will kick in whenever you haven't done anything in thirty minutes

improved some gallery downloader status texts

gallery downloader will no longer wait 5 secs before fetching first page of urls

fixed a critical race condition that had a chance of causing complete and unrecoverable gui freeze in the gallery downloader code

fixed rendering of some bizarro pngs that CV couldn't understand

fixed tag parent management for admin users

fixed up some tag parent dialog code

tag sibling dialog now accepts single reasons for multiple pairs, like the tag parent dialog

improved error state on rendering video that reports resolution of 0x0

removed looping error reporting on certain static image rendering errors

fixed an error when some thumbs were 'collected' while fading in

tagboxes now select on home/end and support numpad versions of page up/down

fixed a radio button initialisation bug

boosted the initial size of the 'set default advanced tag options' dialog so the collapsible pane is more visibile

silenced a upnp error when external ip can't be parsed

reduced some tag dialog previous/next button lag

general code cleanup and refactorisation

gui flag refactorisation and unification

refactorisation of most non-constant stuff from constants files

refactorisations of media mixins class in prep for a general rewrite of that code

improved client initialisation of default options, boorus, and imageboards

version 149

all listctrls with an attendant delete button will now support the same action with the delete key

several normal tag boxes will support the delete key as well

added a temporary service cache to the client db to speed up service fetching until I can figure out a yaml replacement

cleared out inefficient tag and hash database fetching and generation code

tag parent dialog will now ask for an action/reason for every ( children, parent ) pair rather than every ( child, parent ) pair

fixed an error with collecting by rating

fixed the 'fix' to the gui_colours db update problem

the autocomplete dropdowns in the paths-to-tags dialog will no longer produce errors when tags are submitted

improved accuracy of taglist scroll-to-selected scrolldown calculation

collapsed some of the helter-skelter db code in prep for future improvements

moved daemon code to separate files

silenced server upnp errors

all except: continue lines are banished!

cleaned up some miscellaneous exception code

cleaned some ugly behind the scenes a/c dropdown button code

did some general code cleanup

version 148

rewrote thumbnail canvas and scrolling code from a crashtastic monolithic bmp to a lighter, faster, and more flexible page buffer

all custom gui elements should be less flickery

the manage tags dialog will now grow significantly taller if its parent window is also tall

dialogs launched by the media viewer will initially position themselves according to that, rather than the main gui

dialogs launched by controls will initially position themselves according the control's toplevelwindow, as originally intended

thumbnail download (for file repositories) no longer happens silently in the background; it will now occur in the repository sync daemon, reporting its progress in the normal repo sync popup

missing thumbnails are now replaced with the hydrus psi symbol silently, with a simple statement written to the log

fixed a tiny typo in update error code that was reporting version wrong

fixed a typo bug in gettagarchivetags

fixed a typo that had broken namespace sorting

numerous other single-line miscellaneous bug fixes

fixed a bug with displaying media with size (0,0)

fixed a bug with zooming in flash files

improved some buggy tag selection logic that was sometimes desyncing indices between menu popup and selection

tags will now stay selected even through changes to the tags list

any attempt to close the autocomplete dropdown floating frame should now bump the close event up to the whole program

linux release now includes source code alongside executables

version 147

fixed a problem when trying to do a multi-release update that contained the v146 update

fixed the 'canvas not resizing after media removal' bug

when an autocomplete appears in a dialog, the dropdown window will integrate into the dialog (rather than being a popup), allowing mouse interaction

refitted the paths-to-tags dialog so the different static boxes and newly embedded A/C dropdowns fit better

fixes to and cleanup of sibling tag retrieval code

fixes to and cleanup of sibling tag filter code

fixes to and cleanup of sibling tag search code

fixes to and cleanup of sibling tag display code

fixed a couple bugs in READ autocomplete tag search caching that was stopping namespaced queries searching properly if typed in manually rather than pasted

fixed a similar bug in WRITE autocomplete tag search

fixed a bug that was ignoring namespace entirely in WRITE autocomplete

synchronised subscription popups' new file buttons with the text

version 146

manage tags and ratings dialogs will now OK on F3/F4, not CANCEL

zoom switch (default shortcut 'z') will now work for images smaller than the media viewer's canvas

in the media viewer, the canvas-fitting zoom value is now inserted into a media's scrollable zooms

added 'tags box background' gui custom colour

volume, chapter, and page tags will sort properly again in thumbnail view, including last week's improvements

the thread watcher will no longer break its checking loop on non-404 http request or parse errors (i.e. the manual 'check now' button will still work after a misc error)

to remove clutter, the preview window will no longer show tags and file information like the normal media viewer

subscriptions will now show a live 'show files' button as they process

the copy and 'open new page' right-click menu options on tags will now work on more controls

cleaned quite a bit of tags box code

cleaned up and unified a lot of miscellaneous canvas zoom and display code

fixed an error when trying to upload a file petition without any accompanying file uploads

version 145

added custom gui colours for thumbnail backgrounds and borders, the autocomplete background, and media background and text

added <- and -> arrows to manage tags dialog launched from navigable media viewer

on empty input in the manage tags dialog, page up and page down work as shortcuts for the new <- and -> buttons

fixed the media height calculation for animations, so when they are vertically scaled, the total height including scanbar won't overflow off screen

allowed non-integer page, chapter, volume tags in display and sort calculations

semi-integer tags will sort along with integer tags and string tags like so: 0 < 0a < 0b < 1 < 2 < 22

improved the old tag/media sorting code

removed loli and shota from hentai foundry filter options

patched old db-stored predicates to attempt to convert to the new format when queried for _inclusive

this _should_ have fixed the recent export folder problems

created an 8chan board, and updated my various links, including in the client, to migrate from my old forum to this

added a 'fit to canvas' checkbox in file->options->media that will zoom small images to the full size of the media viewer

misc code improvements

version 144

files named 'Thumbs.db' will now be skipped in the import files dialog

fixed wildcard searches, which last week's predicate rewrite broke

fixed a typo that was showing namespace predicates as exclusive (-series:*) when they were actually inclusive (series:*) and vice versa

added wildcard namespace searching for database autocomplete queries

fixed database wildcard autocomplete searching when wildcard match is not the first word in a tag

fixed database file searching when wildcard match is not the first word in a tag

added a comprehensive suite of predicate-unicode conversion tests

cleaned and improved some of the downloader code

added five second per-gallery-page delay to subscription daemon

added three second per-file-delay for regular gallery downloads and subscriptions, just to be polite to those web services

added three second per-file-delay to the thread watcher for the same reason

added a 'check now' button to the thread watcher

fixed a problem that was sometimes causing subscriptions, when paused, to continually restart

removed unnamespaced tag parsing from deviant art

fixed creator parsing for deviant art, which was formerly cutting off the first character

patched an account sync problem in the manage tags dialog

in add tags by path dialog, tags are now sorted before being added to the file list

in add tags by path dialog, the regex sections now generate tags for every match in the string, not just the first

stopped collapsible panels resizing dialogs to minsize on collapse or expand

added shortcut for 'open externally', default Ctrl+E

moved 8chan to new 8ch.net domain (old domain still works)

version 143

when making a READ autocomplete tag query, instances of tags that only have a count in a single namespaced domain will no longer accumulate helper results in the non-namespaced domain i.e. no more 'blah (1)' 'title:blah (1)' dupes

improved the way the above results are calculated

pixiv artist downloading now correctly asks for numerical artist id

reworded some of getting started with files help page to better explain multiple selection

widened the splash screen a bit so db update messages have more space

all frames and dialogs now have an explicit minimum size that is usually far smaller than their initial size

the dialogs with listctrls (import files, manage custom filter actions, and so on) now start a little shorter in height

fixed some bad sizer flags in dialogs with listctrls

rebalanced content update throttling for quicker correction under heavy load

moved the new 5 second break in the gallery parser to a position for quicker status updates

the gallery parser will report total urls found when it is finished (even if that is zero) and wait a bit to let you read that

fixed services->news to hide if you aren't connected to any repositories

autocomplete boxes are now a very slight shade of blue, see if you like it

fixed 'launch file externally' for linux and os x

fixed 'launch directory externally' for linux and os x

fixed 'set up server for me' for linux and os x, I think

improved some general external-process-launching code

cleaned and improved some of the predicate code

version 142

added wildcards to autocomplete results and file queries

autocomplete results will match your wildcard _exactly_

a new predicate type will appear at the top of wildcard queries; selecting it will search files with that wildcard

the new wildcard predicate can be prepended with a minus sign to exclude from results just like normal tag and namespace queries

in wildcard predicates, namespace and/or tag can contain wildcard characters

added some wildcard help

putting in '***' as an autocomplete query is now a pretty bad idea!

fixed some logic in how tags are matched against unusual search input in the db

sped up and cleaned how tags are matched against search input

fixed some namespace logic in how tags are matched against search input

below-character-threshold autocomplete queries will now return all applicable namespace results (e.g. putting in '1' will return [ 1, page:1, chapter:1, volume:1 ], not just [ 1 ])

added 'open externally' to launch the file in default external program to thumbnail and media viewer menu

added a five second delay between gallery-page fetches in the downloader to reduce chance of 429 errors (this was affecting big sankaku searches)

added danbooru webm downloading

fixed a typo in the thread watcher

fixed a bit-rot bug that was stopping the 'like' ratings filter from launching

fixed right click menu in custom filter

version 141

combined mappings are no longer calculated and stored

recalculate combined mappings obviously removed from database menu

combined mappings no longer have to be recalculated on a service deletion; the accompanying 'this could take a long time' warning dialog is gone as well

combined mappings autocomplete counts are now calculated on-the-fly

combined mappings tag queries are now performed on-the-fly

combined mappings namespace queries are now performed on-the-fly

combined mappings zero/non-zero tag count queries are now performed on-the-fly

combined mappings regular tag count queries are now performed on-the-fly

corrected some logic with regular tag count queries, I think

autocomplete tag cache dirties and culls itself more efficiently on tag update

autocomplete tag cache dirties and culls itself more efficiently on file import/delete

removed a couple of non-useful AMP tests that were breaking due to a previous change in connection code

improved how popup messages give the gui permission to update; hopefully the gui will lock up less when big jobs with popups are happening

improved some misc timing logic

improved some repo sync timing logic

added simple emergency throttling to the repo sync daemon when the CPU/HDD is getting hammered

improved some repo sync text number-grammar and timing

added sankaku channel booru, including flash

the booru downloading code that discovers image urls is more flexible

improved my job pause/cancel logic so paused jobs, when cancelled/completed, will report themselves as no longer paused (this was affecting the downloader page, with paused jobs not dismissing themselves on a subsequent cancel)

version 140

if a repository or subscription popup message has nothing to report, it will dismiss itself

fixed handling of text popup display when the object passed was not text

delete orphans is now cancellable

vacuum, deleted orphans, and upload pending popup messages will dismiss themselves an hour after they are done

tightened the subscription final state popup message to just a title and a button

removed much of the very expensive autocomplete tag cache maintenance code, which seems not to be worth the effort

culled the autocomplete tag cache in prep for new maintenance cycle

fixed a resize timing bug that was causing large images to scale in an ugly way when the media viewer was launched in a borderless state

'open selection in new page' will no longer default focus on the sort dropdown; it'll go to the media panel (this was causing scrolling confusion)

fixed a non-updating display bug when resizing frames/dialogs with auto-resizing listctrls on linux

cleaned up a wall-of-text error when closing the client immediately after deleting a tag service

filled a gap in static text image object cleanup

cleaned up some thumbnail waterfall/fade code

filled several gaps in thumbnail object cleanup

version 139

cleaned up all the old popup message code, and fully integrated the new

moved repo sync to the new popup messaging system

moved normal errors and db errors to the new popup messaging system

improved some error handling code

reintroduced message printing

improved subscriptions messaging

added cancel button to:

check file integrity

export to tag archive

added pause and cancel buttons to:

repository sync

subscription sync

pending upload

regenerate thumbnails

improved how jobs' pausability and cancelability are spawned

improved and harmonised a lot of pause and cancel and general shutdown-job-interaction logic

pausable and cancellable popups can only be dimissed with right click once they are done or cancelled

improved some more pause logic in the subscription and repository sync daemons

improved popup pause responsivity

added comprehensive cancelled/finished log statements for the newly pausable and cancellable operations

'just woke from sleep' calculation will no longer evaluate to true on application start

deleted the old broken message log page, which I had forgotten even existed!

fixed a bug stopping adding tag repositories in the manage services dialog

fixed a bug that was hiding the 'name and credentials' panel in the manage services dialog

improved some kinds of thumbnail error reporting

sped up client boot by one second

version 138

created new 'maintenance and memory' options page in the manage options dialog, and moved some things over from 'files and memory', which is now 'files and thumbnails'

added options for idle, vacuum, and delete orphans periods to the 'maintenance and memory' page in the manage options dialog

fixed the incredibly annoying animated scanbar delayed-frame bug, where a click on the scanbar would not draw the new frame until the old frame's expected delay was complete. scanbar scanning is a lot smoother all around, now

animations now show their current frame number in the animation scanbar

status bar now shows inbox/archived counts on any selection

reworked static image zoom code so they resize beautifully, without jaggies, at the cost of a bit of CPU and memory

created new flexible message pathway

moved most messages to new pathway

improved a bunch of message and job_key related code

btw: messages will no longer close themselves; their end state is now to report what happened until you dismiss them. if this turns out to be annoying, I'll change it

fixed the pending menu not updating its count when files imported and added tags via archive sync

improved the logic behind the 'computer just woke from sleep' calculation

improved the accuracy of the 'client is currently idle' calculation

improved database vacuum so the .db-wal file is flushed afterwards, for both client and server

because of better vacuuming, server db backup no longer needs to create a bloated .db-wal backup

removed a very common superfluous empty tag upload packet in normal tag uploads

version 137

eliminated a loophole in the tag-pending process that allowed zero-length subtags through during archive sync

improved the way tags are cleaned and checked, no matter their source

added zero-length subtag (e.g. 'character:') checking and exclusion to the server

removed erroneously pending or current tags that have a zero-length subtag

added 'copy bitmap' option on still image right-click menu

harmonised a whole bunch of clipboard code

added a check file integrity db function, with quick filename existence and thorough content hashing modes

added clear service info cache debug function to database menu

added regenerate combined mappings debug function to database menu

added a primary key to the hydrus tag archive db to save some space on duplicate mappings

added AddMappings to the hydrus tag archive to make mass mapping-adding a bit more efficient

added DeleteTags to the hydrus tag archive

added Get/DeleteMappings convenience synonyms for Get/DeleteTags for the hydrus tag archive

added an 'export to hydrus tag archive' button in the advanced content update dialog

renamed the db tag archive sync methods to better reflect what they actually do

massively simplified how cursors are referenced inside the client and server databases

cleaned up some db cursor code

fixed a critical error in the database test code that was spawning a double db mainloop

improved error handling when regenerating thumbnails

fixed error handling when waterfalling thumbnails

fixed a typo in export tags process

improved the way the client flushes its buffered log data to disk

added log buffer flushing to the server

added a confirm dialog to advanced content update dialog's powerful 'go' button

added custom titles to many co-opted yes/no dialog boxes where the default 'Are you sure?' didn't make sense

cleaned up some misc code

cleaned up some misc bitmap language

version 136

added tag archives

added tag archive sync initialisation on all existing local files

added tag archive sync maintenance on newly imported files

added a new db directory, client_archives

added tag archive sync options to tag services in manage services dialog

added local tags service to manage services dialog

added sha512 to local hash cache

added tag archive hash_type guessing

added a new dialog for selecting n arbitrary strings

got testing to work on all platforms

fixed the hydrus server for linux and os x; it now has a stay-alive frame rather than a taskbaricon

improved some dialog testing code so it would work on all platforms

fixed a deffered problem that was causing the server AMP test to hang on Linux and OS X

neatened and harmonised a bit of common file and network streaming code

improved some misc manage services dialog code

fixed a critical bug that was meaning certain service changes were not being saved to the database, so were being forgotten on restart

fixed some select-string code that wasn't taking sets of strings for a weird wx reason

version 135

added a menu option to any tag's right-click menu to open a new search page for that tag

added a subscription cache to the client database to speed up subs checking

added a filter to the message popup system so those annoying and 99% pointless PyDeadObjectErrors will not display. they will still be written to the log

cleaned up a couple of temporary file deletion errors

improved some more temp file deletion error handling

fixed a bug that I think was stopping file repositories from being deleted

improved the manage services db edit log

fixed a bad comparison that was causing superfluous edit actions after a manage services dialog ok

fixed a new bug related to displaying non text in the popup system

added a 'just woke from sleep' check to all daemons, so CPU heavy stuff like repository sync will not initialise if you just woke your computer. the grace period lasts about ten minutes

retuned the way the subscrption daemon initialises (it'll now wait two minutes after startup before firing)

fixed a typo that was causing fatten service info to fire more often than it should

added a yes/no warning to options ok when the thumbnail dimensions have been changed

added a popup message when thumbnail dimensions have been changed to report on deletion progress

added account testing to my server db testing suite

improved the security of the registration_key->access_key transaction; it'll now generate a new access_key with every call

version 134

updated to wx 3.0.1.1

fixed a critical media scrolling bug due to the wx update

improved some bad media scrolling code, sped things up a bit

fixed 'top' and 'bottom' media scrolling events

fixed a typo that meant the default fullscreen media browsing shortcuts were ctrl+ appended rather than working on their own

overly-verbose errors and other text popups are now cropped to 1KB for gui display, with a notice. the full message will be printed to the log as usual

improved how severe boot crashes are reported

fixed a bit of text-reporting code that wasn't handling non-text very well

improved handling of a weird popup message manager error

fixed an occasional overhasty cleanup error in the checkimportfolders daemon

added options for default values for the thread watcher's number of times to check and check period

fixed the thread watcher complaining about closing when the checking was finished

optimised some id generating code to stop spamming the id cache, which I think was overloading after a while and causing weird PyAssertionErrors

neatened autocomplete dropdown service storage and menu id generation

improved menu id generation for tagbox

fixed opening new petition page from view menu

added a 'this might take ages' warning yes/no dialog when trying to delete a tag service

added a little popup message info to report on progress when deleting a tag service

added some server db testing

fixed an error when double-clicking a tag in a page without search predicates

updated some help links from mediafire to the new github releases page

fixed a typo bug in server db's account flushing code

version 133

reworked the add file process to correct file repository file counts

made add file process to calculate inbox and local thumbnail count more efficiently

fixed a critical bug in server db creation that was stopping file repository service from working

added go/exit fullscreen options to fullscreen right click menu as alternative for default shortcut 'f'

improved fullscreen position, size, maximised, and borderless state memory

removed old 'fullscreen_borderless' option in local options

thread checker now supports 8chan urls

thread checker now has 'number of checks still to do' spin control so you have more options to fire and forget

improved how the thread checker constructs and passes url information around

improved some thread checker timing logic which _might_ have been causing problems

improved some thread checker error reporting

version 132

merged two complicated serverside account tables into two simpler tables

with this action, was able to clean out a lot of rubbish old server account code

made it so accounts can only be on one service. existing (admin) accounts that straddled sevices will have new access keys printed in a text file in the base installation directory on server update

rewrote the account object to be simpler and easier to maintain

swapped the old rubbish 'account_id' identifier in the account object for the much better 'account_key' identifier

harmonised some conflicting account-related variable names

refined the way the unknown account is stored and identified

split serverside account verification and identification into two separate paths, to reduce chance of security problems

reworked account identifiers (a general purpose account identifying object that is used in admin-server interactions) and their associated db functions to be more secure and reliable

simplified account data use checking

simplified and harmonised the way used bytes and used requests are stored and retrieved in the account and account type

with guarantee of account_key uniqueness across entire server, I have simplified session code in several places

updated help to reflect the new relationship between access keys and account keys

added 'copy account key' button to review services, which will now be the thing for users to use if they need an admin to modify their account

fixed serverside credential verification for non-instantiated (still have a registration key) access keys

added a bit of explaining text to the 'waiting' autocomplete state

fixed a typo that caused errors when deleting files from a file repo

version 131

removed tag service precedence and its various expensive and overly complicated effects

recalculated combined tag mappings using new simple union based merge

cleaned updatemappings code, and I think fixed some tiny count logic bugs

refactored eight overly complicated sub-functions of updatemappings down to two simpler ones

fixed up a mention of tag service precedence in help

deleted a bunch of old client update code

added adaptive throttling to tag updates, so they'll speed up and slow down based on current CPU load

refined a bunch of the parent and sibling code, given the new changes

updated and cobbled back together a host of finicky parent and sibling and regular tag test code, given the new changes

fixed a bit of logic in the censorship code that was causing overly broad namespaced matches

fixed error when selecting a specific tag service in a normal search

deleting a service no longer throws an extraneous error after it is complete

fixed danbooru large pngs, which were being resized in a way that broke some of the downloader's url prediction code and caused 404s

fixed a bug in the input custom filter action dialog

fixed a misc clientserviceidentfier bug in custom filter initialisation

fixed setting numerical ratings from a custom filter (I think) and I think fixed many other custom filter operations

fixed several text-entry-dialog-launched-from-dialog bugs

fixed, I really think for real this time, the serverside account desync bug. it was in session init, where I had not previously looked

improved some popup messages' text

improved the way some exceptions are reported

cleaned some error reporting code

silenced the more-annoying-than-useful account sync failure popups

fixed and cleaned server start/stop/restart code a bit more

reduced regular CPU hit from resize thumbnail daemon and made it more polite in starting

version 130

fixed youtube downloads

fixed remote services not showing in OS X manage services dialog

updated 'running from source' help page with some new info for Linux

cleaned a little more code

fixed bug that was stopping new/modified serverside services booting immediately

improved the server-client service key translation code on serverside service modification

fixed test credentials button in manage services dialog for unsaved services

fixed bandwidth tracking errors for unsaved services

deleted some old unused server options code

improved server services start/stop code a bit more

massively rebalanced the autocomplete maintenance routine after a tag service deletion or tag service precedence change

made a mess of some of the deleted tag update processing code, I'll fix it next week

version 129

fixed a typo in launching the ratings dialog

did some important code cleanup re a common ambigious variable name

improved serverside account tracking in a way that will help future user management clientside too

improved serverside session code a bit more

improved general serverside login

improved serverside access key verification

aligned server db update code with the client's only-update-up-to-50-versions rule

fixed some buggy repository test code

fixed some buggy local booru test code

version 128

fixed right click on thumbnail error

improved assumed permission defaults on tag dialogs and right click download/upload actions (i.e. tag dialog's entry A/C will no longer disappear when you desync from tag service)

removed account desync on network version mismatch, to stop this problem anyway

fixed tag parents not showing in autocomplete dropdowns

added sqlite3 command line tool to db folder for convenience

fixed a typo in manage server's services dialog

some misc code cleanup

added default advanced tag options pane to manage options dialog

added default advanced tag options to manage subscriptions dialog

fixed booru namespaces in manage subscriptions dialog to be booru-specific

added default advanced tag options to normal download pages

version 127

finished service_identifier rewrite, phew

added an exception catch for full program crashes; big errors will now be written to crash.log

added a shutdown check to make sure popup message manager closes properly when certain weird errors occur

'files_service_identifiers_cdpp' is now renamed to the vastly less insane 'locations_manager'

removed db update code older than 50 versions

a new error message will display if trying to update more than 50 versions in one go

many other small fixes I didn't keep track of

version 126

restored a dll that I thought was no longer needed, but was actually doing some weirder gif rendering

added 'remove' to fullscreen menu

harmonised thumbnail and fullscreen right click menus a bit more

added pause button to popup messages for repo update and subscription processing

moved service_identifier switchover forward

moved all service fetching to streamlined and non-laggy manager

changed client options to store default tag repository in a better way

changed subscriptions to store their advanced tag options in a better way

fixed a 'missing service' bug in advanced tag options

remade idle calculation into a much better gui-based rather than db-based test

fiddled more with maintenance timing, hopefully for the good

new screenshots in the help index pages!

improved how auto repo and server setup work and report their status

the client's UPnP daemon will no longer spam errors if your IGD doesn't support UPnP

fixed a bad db call in server's UPnP  daemon

version 125

moved client splash screen and client boot to application event loop (i.e. your mouse won't hourglass over it now)

splash screen is now draggable

added splash screen for client shutdown

cleaned up shutdown procedure

added detailed A/C timing options

created a new text entry dialog that won't allow ok on empty text

hence fixed a startup bug when a session was created with no name (and likely others!)

further tweaked idle calculation to improve maintenance routine

revised a bunch of the maintenance timer logic

fixed some bugs in the maintenance timer logic

fixed a daemon error reporting typo

fixed another way session data usage tracking could split, server-side

brushed some more dust off the server-side session code

improved some bad image processing nomenclature

took first steps in service_identifier replacement rewrite

version 124

fixed some more broken gifs (those with funky transparency, I think)

fixed a critical bug in the subscriptions dialog that was causing subscriptions to be deleted

reintroduced old locale code, with minor improvement, to fix number grouping (123,456,789) issue

added a short pre-activation wait to all daemons to reduce more wake-from-sleep problems like the UPnP issue last week

version 123

rewrote the old gif rendering code to the new system, so gifs with variable framerates now work again!

fixed a bug that was stopping the first zoom of an image from showing if it took more than a frame to render

webms are now zoomable beyond frame border

webms will now respond to keypresses and mouseclicks when cursor is inside their border

added service select to import metadata

if repo sync is paused, either globally or individually, repo will no longer try to regularly sync their accounts

fixed individual repository pause sync

I think I fixed a serverside session synchronisation bug that was causing too-small data usage reports

improved complicated tag action descriptions in manage tag dialog

hastened initialisation of subscriptions daemon

hastened initialisation of subscriptions dialog

neatened all object declarations

fixed a menu bug in OS X

added put a little delay in the upnp daemon, hopefully to stop the error popups after waking from sleep

updated an old hacky bit of locale code that was breaking certain platforms of wx 2.9.5.0

improved animation and other rendering timings on non-windows platforms

improved media focus on non-windows platforms

version 122

fixed server upnp daemon to use correct port, I think

added client upnp daemon for local booru; it'll update upnp mappings as soon as they are changed

added bandwidth tracking for local booru; it'll even magically update before your eyes in review services

bandwidth limits are tested, so exceeding data usage will result in 403 errors until the next month

moved local booru css to an external file any user can edit

added 'open share in new page' button to local booru review services

fixed local booru display for video, flash, audio, pdf and miscellaneous

fixed local booru render resolution for all media

fixed local booru thumbnails for video, flash, audio, pdf and miscellaneous

added caching headers to static file requests to reduce browser-hydrus bandwidth

fixed local booru page text's newline formatting

added local booru unit tests

version 121

first version of local booru

version 120

improved quality of downsized animation rendering

sped up downsized animation rendering in this case

neatened animation code

fixed a mid-animation resize parameter bug

I think I fixed an animation scan bug that was sometimes giving the wrong frames

added thumbnails for all video formats

fixed an off-by-one framecount bug for certain videos, and retroactively fixed counts for existing videos

fixed a couple harmless width/height numpy.shape switcharounds

cleaned up some file parsing code

added a pixiv unit test

fixed a bug when servers were returning unusual relative redirect urls (gelbooru)

semi-fixed a bug when servers were returning unescaped redirect urls (gelbooru)

version 119

fixed an overzealous error in v118 update code

new direct ffmpeg video decoder for webm

new system takes advantage of multi-core processors, increasing render speed

improved animated frame timings

fixed a number of gifs that were causing the "None has no 'shape' attribute" error

sped up memory and subprocess cleanup after video playback

added some new ffmpeg instructions to the running from source help page

added client and server startup and shutdown statements to the log

improved some idle time logic, which should improve maintenance timings

fixed a critical idle timing bug that was adding 1s to many actions, including file queries!

fixed a pubsub testing bug that was preventing daemons from quickly closing after a unit test

version 118

improved animated frame buffer size calculation, saving memory and improving smaller animation performance

improved quality of static image thumbnail generation

fixed webm mouse event propagation

updated and improved perceptual hash code

version 117

fixed the gif rendering error that was causing threading lockup

fixed a bug in the animated drawing code that was causing +5ms duration on every frame

improved frame duration checking code

lowered quality of animated frame resizing to improve effective framerate for previews

fixed a problem with pixiv session management

fixed a problem with pixiv tag subscriptions

neatened an error with pixiv manga download failures

corrected reviewservices hyperlinkctrl initialisation

improved the wording for the 'port already bound' error message on client startup

improved hydrus network session error handling

cleaned up some old gui messaging code

some general image/video code nomenclature cleanup

changed http connection maintenance daemon to be snappier on client shutdown

subscription minimum check period is now 1 time unit, not 0!

version 116

gifs now render their frames as needed, rather than building a giant memory-hogging cache of all of them

webms now have thumbnails!

refactored important hydrus threading code to a separate module

added new 'call-to' worker threads for snappier job-thread behaviour

fixed a typo that was stopping new repositories being entered in the manage repositories dialog

version 115

moved review and manage services windows to a local/remote dichotomy

moved local booru service code generally forward

added local booru service to db init/update

added booru to review services

added booru to manage services

neatened some misc client code

improved static/animated image rendering code in general

added some cv stuff to gif rendering code

cleaned up gif rendering code

sped up animated gif rendering by about 5-10 times!

added some cv stuff to static image rendering code

cleaned up static image rendering code

sped up static image rendering up to 4 times!

image import should generally be faster

pulled animated images off the image cache, reducing memory usage

added some clever compression that cuts image cache memory usage in half!

version 114

gif rendering seems to be fixed in all cases! hooray!

fixed 'pop from empty list' popup error spam in the new cache clearing system

fixed weird behaviour on right-clicking 'dismiss all' popups button

added a unit test for perceptual hash generation, with an eye to moving the delicate code from cv to cv2

updated opencv to 2.4.9

updated pyinstaller, so frozen releases may be a bit more stable!

moved webm and mkv info parsing over to cv, which allows num_frames

version 113

added mkv+webm support!

fixed some system:mime selection stuff

fixed a missing db call in the server upnp daemon

fixed upnp dialog's edit button

cleaned up upnp dialog's post-button refreshing logic

fixed getupnpmappings to deal with >10 mappings

fixed fullscreen_switch for custom filter

fixed a late canvas drawing exception when fullscreen window was closing due to being media-empty

rolled back to wx 2.9.5 for windows again. not worth the trouble, right now

hydrus caches should now clean up anything older than twenty minutes

'last session' will now be saved every five minutes, rather than only on a successful close

version 112

A/C dropdown is fixed on linux!

wrote new UPnP library based on miniupnpc multiplat executables

integrated new UPnP library into existing client code

upnp management dialog should now work for os x and linux

integrated new UPnP library into existing server code

added UPnP unit tests

fixed a bug when upnp executable path had whitespace

renamed 'add, remove or edit services' to 'manage services'

fixed some review services frame text display issues for restricted non-repository services

optimised review services frame initialisation db requests

added a database maintenance routine that should help review services launch a bit faster on average

improved manage services layout a little bit

removed initialise server button when it is invalid

updated 4chan thread url parser to handle new 4chan thread urls

redirected 4chan thread downloader to new api address

fixed an error reporting problem in repository sync

fixed my linux dev environment, so linux executable is back

changed wx requirement to 2.9 on linux, due to bugs in linux 3.0.0.0 release

fixed a typo in repository  deletion

added repository deletion to unit tests

added num_base and num_step to file path regex dialog

fixed popup display of certain exceptions, including file parsing exceptions

cleaned up os x and linux build scripts, added static&help sync

version 111

fixed booru searches that include unusual characters like '&'

added pause synchronisation to individual repositories in the manage services dialog

fixed gui_sessions with predicates (again!)

added a unit test for gui_session storage, shouldn't get problems with that again

fixed a getcountlesscopy predicates bug

first step in multi-platform upnp support is DONE, phew

harmonised subscription file import errors with normal hdd import errors. they nshould be a bit clearer

corrected how popup text width is enforced

added a warning to the reset cache button in the manage subscriptions dialog

os x autocomplete dropdown seems to be magically fixed, not sure how or when that happened

started HydrusBooru

fixed some problems with twisted startup/shutdown in unit testing

version 110

fixed a variable overwriting issue in the subscription daemon error reporting code that was causing error spam

fixed more actual and potential instances of this error elsewhere

fixed a bug in the import file error system for non-local files

fixed url parsing for urls lacking 'http://' or 'https://'

fixed hentai foundry image parsing for two different cases

fixed client gui-initiated db backup to also backup client_updates folder

A/C dropdown now shows current and pending counts separately!

fixed display of uncaught database exceptions

new version of sqlite for windows, might speed some things up a bit

upgraded a bunch of other libraries, look forward to mysterious new bugs!

I _think_ some transparency support is improved

moved messaging forward

dropped processed_mappings experiment

removed a bloated mapping index, hopefully speeding a couple things up

fixed an erroneous error message in hydrus network session manager

version 109

started processed_mappings table. for now, it mirrors normal mappings table

improved manage tags dialog logic

fixed only_add issue for parent tags

fixed an important tags box surplus tags display bug when focus repository was changed

fixed petitioned tag counts in tag boxes, as long as 'all known tags' is not selected

fixed occasional zero tag count in tags box

added petitioned tags to normal media fetch, not sure why they were missing

fixed a random error when trying to refresh a 'open selection in new page' page

improved help to reflect new manage tags dialog

cleaned up an annoying spam-text-to-console issue when running test.py

reworked the way the server clean/create update daemons interface with the database

improved how updates are stored and fetched in the server db--no longer needs a db hit to fetch

harmonised client db's update naming convention to the server's

improved server db to update the same new-ish way client db does it

version 108

added 'database->delete orphan files' for manual firing of this maintenance routine

improved redirect location parsing--should fix the booru 'Could not connect to None!' errors

fixed 4chan thread downloader's erroneous 'still working' warning when trying to close while paused

added multiple additions to manage tag parents dialog

added multiple additions to manage tag siblings dialog

generic improvements to both dialogs

manage tags dialog now shows all the selection's tags combined, not just the tag intersection

multiple improvements to manage tags dialog

reworked some tagsbox classes to be more flexible

improved dialogsetupexport's use of tagsbox

version 107

converted 'namespace blacklists' to more general 'tag censorship'

tag censorship now accepts unnamespaced tag bans

tag censorship now accepts all namespaced tags bans

tag censorship now permits bans for all tag services

fixed several small bugs carried over from the namespace blacklists code

added a bit about tag censorship to help, under advanced.html

fixed youtube downloading

file import dialog now ignores dupe paths

improved update status text in review services window

fixed erroneous capital P on pending menu

fixed shortcut input dialog's ok button

fixed modify tags dialog's admin modify button

added several bits of explanation text to advanced dialogs

added more helpful timestamp to news dialog

neatened export dialog's layout

corrected some double-escaped backslashes in the export filename pattern button menu

fixed modify accounts dialog's http request

improved grammar in db pubsub code

version 106

download by raw url now sends the correct text to the popup on successful import

first step in peer-to-peer repository sync is complete

repository updates are now stored as local yaml, in the new client_updates folder

repository reset keeps the downloaded updates, only does a reprocess

all repository updates should process faster, especially if you have saved sessions

a lot of misc gui lag should be reduced, especially if you have saved sessions

the way the gui's menubar is updated is massively improved

os x and linux menubar should be less spastic

several improvements to daemon pubsub workflow

version 105

complete rewrite of message data handling and display

hidden gauge messages will now display correctly

fixed error-display bugs in the log page

massively improved how database errors are generated and displayed

improved and streamlined how other errors are generated and displayed

improved the way all gui elements are deleted from memory

fixed the frequent segfault in os x and linux when closing popup messages

fixed the common segfault in os x and linux when refreshing a query

fixed the occasional segfault in os x and linux when closing a page

fixed pattern shortcut buttons

the 'show deleted' checkbox on manage tags dialog makes a return

neatened several checkboxes so clicking the label text clicks the checkbox

version 104

first version of export folders is ready

cleaned up all ok and cancel dialog behaviour

cleaned up some misc dialog button code

fixed up some ugly db yaml pubsub stuff

fixed a juddering 100% CPU loop in the subscriptions code

subs and repo sync will now put up a popup gauge

repo sync will now pause mid-update on repository pause

services->pause->stuff will poke daemons to restart more reliably

removed the service status area of the statusbar

fixed 'subscriptions forgetting their tag options' bug

added select invert, thanks to @fluffy_cub for contribution

fixed a slight logic bug with display of select archive/inbox

added better instructions on how to run in OS X from source, thanks to user steenuil for contribution

version 103

fixed a virtual size issue with expanding collapsible panes inside scrolling windows :S

fixed images not showing on first zoom switch

improved touch and feel of image/flash scanbar

subscriptions dialog is now flattened

reworked advanced_tag_options to be more efficient and flexible

fixed a couple bugs in advanced_tag_options

moved fake http unit tests to a fake version of the new networking system (so much better, now!)

moved real http unit tests to new networking system (so much better, now!)

fixed up a last couple network things

I probably fixed some file upload code, as well

cleared out old network code

fixed a fake pubsub problem in unit tests

made a single new db table for consolidating:

4chan_pass

pixiv_account

boorus

favourite_custom_filter_actions

gui_sessions

imageboard_sites

imageboards

import_folders

subscriptions

couple small help edits

fixed view->new booru download page

started export folders

version 102

initial linux release

rewrote running_from_source.html

altered some service object and db stuff to look after null access keys and temp services a bit better

key registration works on new network system

all downloaders use new network system

A/C timer tweaked a little more

did some timer event grammar improvements

improved button behaviour in import files dialog

backing up a server no longer causes a timeout problem

prototype export/import tag metadata is done

rewrote the way animated gif frames are navigated and thrown to screen

fixed an issue that was causing some dialogs to OK when escape was pressed or the close cross was clicked

this above issue included the enter system predicate dialog

version 101

fixed another update problem for clients <v95

uploading pending content should be less gui-blocking

fixed numpad navigation of page chooser to work for qwerty rather than dvorak as default

os x page picker now obeys arrow and numpad keys for navigation

os x command key should be treated as ctrl now, for better cross-platform compatibility

a couple more cmd/ctrl confusions cleared up

os x menubar disappearing after fullscreen problem fixed

added stateless http connection manager

simplified how http works in hydrus

http connections now used more efficiently

better cookie parsing

better http application shutdown handling

some misc http bugs fixed

new http error handling should limit occasional spam-errors

moved all hydrus http connections to new manager

moved many non-hydrus http connections to new manager

version 100

MVC import_controller system created

download gauges lag reduced

download cancel button desync bug fixed

download error code harmonised

download async code pushed to threads

download job key checking and management improved

download queue feed timer will obey page-hidden pause status

download success and exception handling harmonised

download on-close thread-signalling much improved

download pages now take responsibility for download code factories

download management panel code harmonised

download management panels made much more MVC

download error reporting is much improved

download fail management is much improved

download button response is much faster

download progress display is split into the easier to read file/queue/builder trichotomy

download display indices are less schizophrenic

lots of misc improvements to download code and nomenclature

thumbnail page's refresh and select menu items now only appear when appropriate

thumbnail page's select menu will only show inbox/archive sub-items when appropriate

thumbnail page's menu is less buggy generally

db updates are now parcelised into each version, rather than one big job

improved db version update notification

a problem with updates from ~v70 to >v95 is fixed

fixed a bug in popup message dismissal

database exception formatting improved

database exception display spam reduced

database exception redundant traceback removed

autocomplete character-search-delay time extended from 150ms to 250ms

async wx window destroy code improved in certain places

improved non-wx-thread popup error reporting

some other bugfix, grammar and nomenclature stuff I can't remember

version 99

added backup database menu option

added restore database menu option

gone back to wx 2.9.5 for now

fullscreen scrolling should be a bit smoother now

fixed sessions for 'open selection in new window'

new regex shortcut for filename in the 'add tags before importing' dialog

fixed dialogs cancel-closing on an unfocused 'enter'

finally fixed the animation scanbar vertical sizing on preview screen

animation scanbar now shows gif frame rendering progress more smoothly

test.exe now included in windows release

some frame parents code cleanup

a tiny bit of gif code cleanup

deleted some old server code

cleaned up import pause/cancel code

improved hydrus's thread-communication objects

cleaned up downloader code and logic and gui display

version 98

update to wxpython 3.0

you can now add tags when importing from a zip

you can now download arbitrary urls from download->a raw url

'select files to import' dialog has massively improved file parsing, with:

new gauge location

queueable parse jobs

paths that stream as they are parsed

pause/cancel buttons

new 'check credentials' button in manage services dialog

new access_key_verification server request

fixed a very important bug that affected normal tags when adding parents or siblings

fixed a bug where old pending parents would not be remembered when new same-child parents were later pended

fixed a similar bug where rescinding parent pends would screw things up

pending counts will new correctly update on a sibling or parent pend rescind

parent tags no longer (incorrectly) remove when:

removing 'just this file' child tags in 'add tags before importing' dialog

removing child tags in custom filter

some youtube videos were not downloading due to a redirect issue. it is improved, possibly fixed completely

collapsible panes, like 'advanced import options', will refit dialogs on collapse/expand

fixed an bug in the way certain server errors were handled

fixed a serverside bug for GET account when using account_id as identifier

fixed 'modify petitioner' button in petition pages

fixed a bug that was preventing clearing of temp path on client startup

cleaned up some credentials code

some general code grammar improvement

reworded some help

reworked account identifiers

os x

now has an app icon, woop

os x page closing logic improved

collections checkbox dropdown seems to work, now

fixed segfault on popup message close

thumbs loading from initial session seems to draw properly, sometimes

fullscreen no longer causes segfault whenever you zoom or scroll

version 97

initial os x test build

collect typo fixed

'remove all mappings with specific namespace' added to advanced content update dialog

version 96

went over _all_ my help (about 200KB of text!), cutting down waffly paragraphs, rewritting bits that were stiff or unclear, updating screenshots, and adding several new sections people suggested

did a complete rewrite of how the client stores services in its database, streamlining everything

several bugs exposed by the services rewrite fixed

several moments of weirdness exposed by the services rewrite made more sane

new version of sqlite (3.8.2)

new version of python (2.7.6)

cleaned up a debug print statement that was spamming gauge numbers to the log

when loaded from a session, hdd import pages will now hide their processing panel

fixed namespace sorting to manage numbers again

collapsed the many different download gallery page classes to a single one

started an important rewrite of how downloaders are going to work in future

improved testing framework's pubsub testing support

subs sync daemon now restarts correctly after a dialog-driven pause

parents now obey siblings!

siblings and parents are updated on forget pending

got AMP message between persistent and temporary peers working

version 95

pages launched from a session will no longer run their query on startup; they will remember their thumbnails

import pages added to gui sessions, although they do not yet remember their progress

the way session pages are started is improved

advanced content update dialog simplified and rearranged

advanced content update db code written for copy operation

advanced content update db code written for delete operation (local service only)

advanced content update db code written for delete deleted operation ( local service only)

made first version of IMMessage objects

added network version checking to hydrus's AMP

better AMP error catching and reporting added

AMP tests moved further forward

tightened up AMP network code, fixed last bugs

cleaned up a huge pile of 'expiry/expires/expiration' semantic mess

same for some mapping_ids/mappings_ids stuff

removed predicate knowledge from media display classes, which was just a mess, really

corrected a weird param order in sessions

general code cleanup

several old instances of tuple-list yaml compromise removed

fixed a session bug in the testing code because of last week's manager management change

fixed a hydrus session bug for accounts with no expiry

fixed a very important server-side network version checking bug

added client-side network version checking

version 94

reorganised the 'add tags from path' dialogs so the regex buttons made a little more sense

when deleting a file, the current focus will pre-defocus

'confirm close client' dialog will now auto-yes after 15 seconds

fixed a bug when POST hydrus queries couldn't return any data

'failed to update repository' message made simpler

added tuples to pyyaml's safe_loader (which was more of a pain than I expected)

manager management improved, with changes reflected across all the code

session management prototype is done

query pages can be added to sessions

default startup session can be set in options

sessions are managed with new submenu under file

creating registration keys will show them in a new, more helpful frame

same for initialising a server

and for auto server setup

auto server setup has cleaner error reporting

new service admin accounts now have different access keys to the server admin account

fixed up testing framework to test AMP a bit better

IM session manager written

server db support for new session manager added

version 93

can now undo/redo all tag operations except rescind petition

can now undo/redo file pending, rescind pending and petition

improved undo/redo description generation

fixed a bug that was showing extraneous undo/redo actions that did nothing

review services frame's permissions strings will size a little better

new 'perform a service-wide update' button on review services frame

started tag-specific dialog for this new button, and began db code that'll execute its stuff

a little code reorganisation

fixed a bug in the export dialog that was throwing errors when trying to overwrite read-only files

moved a number of old dialog error messages to the new verbose popup system

when accounts are made stale (as part of client version update), they will no longer lose their privileges

fixed the log page to deal with (ignore) gauge messages

fixed a db problem in my testing framework

AMP IM support moved way forward

IM manager first draft done

version 92

encrypted instant messaging framework started

encrypted instant messaging test added

encrypted instant messaging trust framework started

started an AMP framework for encrypted instant messaging

tag A/C exact match parameter added

tag A/C exact match parameter test added

tag A/C logic reorganised and improved

tag A/C now gives exact matches for queries shorter than the A/C threshold

tag A/C threshold default set to 2

fixed an important architectural bug in pubsub

generally cleaned up pubsub code

'popup messages sometimes not updating' problem is fixed

a bit of initialisation cleanup to make startup behaviour more reliable

version 91

improved how accounts are identified in the server

rewrote how serverside session data is managed

added monthly data reset to session data

fixed session update on account modification

fixed a bug in accountidentifier generation from accounts

fixed a bug in account-multi-session tracking

server session manager unit test added

reworked testing to take arguments to only run specific tests

changed all file downloads to be more memory efficient, either by gui or subscription

made download testing fail better

added deviant art unit test

fixed deviant art thumb parsing error

fixed deviant art fullsize image parsing error

undo system reworked in prep for first prototype

undo/redo keyboard shortcuts (ctrl-z and -y) infrastructure added and db updated

undo works for fullscreen

a bunch of fullscreen keypress processing improved

undo manager testing added

added upnp option to all services

added upnp daemon to maintain upnp serverside

'copy all tags' and 'copy all tags with counts' added to tagslists

rejiggered a bit of the app startup, including db password check

found a better error noise

in fullscreen, the mouse cursor will no longer auto-hide when over an animation scanbar

version 90

client db unit tests added:

4chan_pass

autocomplete_tags

booru and boorus

downloads

favourite_custom_filter_actions

imageboard

import_folders

md5_status

media_results

namespace_blacklists

news

pixiv_account

improved services

sessions

shutdown_timestamps

fixed modify account

fixed modify account again!

made account_info respond a lot faster with large tag counts

neatened some client db code

youtube 'file already deleted' error now reported correctly

youtube general error reporting improved

fixed multiple-computer-single-account data use reporting

moved file parsing from nested dialogs to a single popup message gauge

fixed an export folder path-display issue when an install moves location

improved how some file downloaders process their files

added prototype file download status bars to all download management panels

fixed a bunch of tests that were cleaning up inelegantly

version 89

fixed blacklist manager for numtags

fixed a sql typo for numtags

delete from fullscreen menu fixed

fixed a unicode error-handling issue

sped up autocomplete tag fetch for large result sets

improved the way autocomplete dropdown scrolls

vastly improved efficiency of how all listboxes draw

the server will no longer start if something is using the server admin port

i.e. you can no longer run multiple copies of the same server by accident

improved client and server port change (service shutdown and restart)

improved testing framework to handle fake db reads better

improved testing framework to manage client and server sessions

several network request unit tests added:

access_key

account

account_info

account_types GET

account_types POST

file_repo file

file_repo thumbnail

init

ip

news

petition

registration_keys

services GET

services POST

session_key

stats

update GET

update POST

fixed clientside permissions exception

several db unit tests added:

import_file

system_predicates:

age

archive

duration

everything

file_service

hash

height

inbox

local

mime

not_local

num_tags

num_words

ratio

similar_to

size

width

limit

fixed a bug in system:age?

fixed a bug in system:duration<0

fixed a bug in system:duration?0

improved num_tags logic

fixed system:num_tags<1

fixed system:num_tags>1

fixed a hash filter bug for system:similar_to

fixed system:similar_to for unknown hashes

fixed system:size Bytes

updated linux.html and running_from_source.html in help

version 88

moved server daemon start to a better place

cleaned up more server db code

fixed getnumpetitions

fixed getpetition

reworked modifyservices and fixed server request

reworked modifyservices dialog to work with new system

server service options now managed from modifyservices dialog; old options dialogs deleted

auto-repository setup restored and fixed

harmonised edit_log actions all across the code

added namespace blacklist manager

added namespace blacklist dialog

integrated namespace blacklist into:

search by tag

search by tag count

creating media results (thumbnails)

autocomplete tags read

autocomplete tags write

fixed a problem with deleted tags not being applied correctly on initial thumbnail display

started a server unit testing framework

added 'running in linux' info to help

version 87

misc:

fixed system:untagged, which was doing numtags>0 by accident

basics:

moved client local service to twisted

moved server to twisted

cleaned up a whole lot of unrelated server stuff I haven't touched in a while

fixed some misc typos

details:

built twisted server framework to plug into hydrus

changed hydrus authorisation header

changed sessions to manage accounts, reducing server db load

reworked session cookie to be neater

cleaned up a bunch of server code

fixed how certain server errors were being printed to the log

reworked response_context to a body/path dichotomy to reduce cpu+memory on file requests

collapsed a bunch of GET POST requests to the same path

reworked server_service_identifiers to be port-independant

improved data use logging to support new session management system

moved session management to new system

moved error management to new system

integrated twisted thread into hydrus controller

reworked root and favicon

reworked local file and thumbnail requests

reworked restricted service requests

reworked admin service requests

reworked repository service requests

removed manage options query; it'll be rolled into manage services admin queries

updated serverdb to manage with new system, harmonising internal and external requests to one workflow

made server-side data use tracking simpler

added Content-Type header to most requests

improved how key registration, init, and account GET requests are handled client-side

harmonised how account_keys are created server-side

moved server management from serverdb to servercontroller, with better pubsub restart

moved several useful functions to the new serverconstants.py

fixed a max_age sessions issue

version 86

timeout on connections improved

rewrote the objects behind the dumper

the dumper should now select media properly again

closed pages now timeout after an hour

upgraded to new version of sqlite

fixed client port bind detection and error reporting

you can set the client's local server's port in client options

local server port changes will happen when dialog changes

unified the upload and youtube download popup messages into one popup

fixed changing thumbnail size (stupid typo!)

added 'database->regenerate all thumbnails' to fix certain thumbnail errors

add a popup gauge for regen all thumbnails

fixed yesno dialogs, which were showing a neutral 'Cancel' button rather than a red 'no'

the upnp dialog's buttons now work, have fun with it!

version 85

updated to new version of wxpython

several key_down events moved to char_hook

rewrote collapsiblepane class to remove weird refit and scroll behaviour

a problem with adding service via a registration key is fixed

tag upload pending is granulated into separate queries, to reduce server and client lag

uploads moved to an asynchronous popup message with a gauge, rather than the modal progressdialog

reorganised file upload content_updates to be a bit smoother

fixed a redirection bug that wasn't obeying schema changes such as http->https

updated e621 url to https schema

version 84

switch fullscreen button added to fullscreen canvases

messaging stuff is disabled for now

gif scanbars now fill up as the frames render

pngs with transparency are no longer drawn with black background after first viewing

made a small change to listbooks to correct some gui-weirdness, particularly in local options

autocomplete dropdown windows will now hide-and-reposition on parent scroll events

autocomplete hide-and-reposition waits 250ms rather than 100ms, making it a little less flickery

a graphical bug related to hitting end on a large search is fixed

upnp framework started

services->manage local upnp started

new messagegauge added

fixed mp4 import

youtube url->formats added

youtube format chooser dialog added

youtube downloader thread added

youtube gauge popup added

youtube error handling improved

youtube num_bytes_done added

youtube unknown total_num_bytes handled

youtube cancel button added

youtube right click dismiss throws up a yes/no dialog to decide whether to cancel the download

started manage upnp dialog

closed page undo added

closed pages will be paused and quiet

pause import folders option added

fixed an invalid index drawing bug after removing certain media

thumbnail waterfall improved

tags are limited to 1024 characters

version 83

sort by longest fixed for files with no duration

reverse sort by rating fixed for files with no rating

sort by largest, newest fixed for files with unknown size or timestamp

sort by unknown/absent values sorts more accurately

search by num_tags and min_num_tags fixed

remove media thumbnail update bug fixed

currently viewable indices screwed up after a remove media bug fixed

remove all media entire black screen bug fixed

delete media redraw optimised

several message print unicode/raw byte errors fixed

scrollbar position calculation improved, so 'black rectangle on scrolled thumbnail canvas when num_cols changed after resize' bug should be fixed

'tiny black lines on small slider drag' bug fixed

moved a bunch of right_up events to right_down, see what you think

popup message cleanup is better on shutdown

pubsub typeerrors handled more gracefully

improved how options init and update works

client will now remember the restored size, restored position, maximised state, and screen the gui was last left at

client will remember all the same details for fullscreen, separately

these sizes _should_ rescue from offscreen if you disconnect/reposition a non-primary display

shift+home/end now scrolls and selects, rather than only buggily scrolling

I fixed a content_update bug re petitions

'show in new page' pages will no longer show search subbox

added hentai foundry tests

fixed hentai foundry title and creator tag parsing in certain cases

fixed some related html unicode issues

initial value on tag siblings dialog fixed

tag siblings and parent dialogs now start sorted by the right column (parent/older sibling)

setting a default collect will no longer cause a nasty crash!

setting an orphaned default collect will no longer cause problems

deselect thumbnails has improved focussed media management

can now select inbox or archive from right click menu

version 82

a bug where slow search results would sometimes appear after search predicates were removed has been fixed

a lot of autocomplete gui- and db-blocking reorganisation

searches are now entirely asynchronous to gui thread

searches are split into two granular phases, and are cancellable during processing

simplified system predicate storage

consolidated all system predicate filtering to initial db search

huge improvements to how the thumbnail canvas is sized and extended and drawn to

numerous fixes and improvements to how thumbnails are drawn to screen

reworked how collect works

and sort

the raw cpu time behind sorting in muuuuuuuch faster

clarified my listening media class

added another optimisation to canvas resizing

big improvement to canvas redrawing and refreshing

important fix in how client figured out what to draw when clicking in whitespace

thumbnail fade is much smoother

thumbnails should now generally draw a little smoother

selectall and selectnone's thumbnail fade is less cpu intensive

cleared up a number of collect related selection and index bugs

fixed a media identifier issue

all import-related append and drawing is muuuuch faster

flicker bug on small appends fixed

fixed a page up/down drawing bug on the main thumbnail canvas view

fixed an unknown-timestamp thumbnail right click issue

made my internal media data storage system a lot simpler

new sortedlist class to make some media stuff easier to manage

reworked how new media is added to a page of thumbs (usually import-append, but the new system supports collect- and sort-sympathetic insertion):

-calculating combined tags on the left can be up to ten times faster

-integrating new files into the sort is much quicker

-integrating new files into the collect is much quicker

-improved how new thumbnails are decided to be drawn

trying to archive or inbox more than one file, you'll get a yes/no dialog to confirm

started db testing framework

db tracebacks improved

slimmed down content_update data-side processing

a couple small code fixes

archive and inbox no longer remove from the search if the opposite system predicate is set

system:rating fixed

fixed a bug where you could middle-click-download files that had no known source

couple of bugs with system:ratio fixed

system:ratio string representation is 16:9, rather than 1.777779

gui thread is more intelligent in telling non-gui threads that it is busy

version 81

mp4 added

mp4 mime search added

wma added

wma mime search added

wmv added

wmv mime search added

changed to video and audio icons, rather than one for each mime

popup messages now dismiss on RIGHT_UP rather than RIGHT_DOWN

moved many more messagebox errors to the new popup error system

cleaned up clientgui's superfluous error handling

rewrote error redirection to new popup system

more error overhaul

print statements overhaul

better db updated statement, through popup system

traceback show/hide button added to new popup error box

copy button added to new popup error box

added a 'x more messages/dismiss all' popup when there are more than ten messages to display

review services refresh account button will disable until the server replies

file system predicate dialogs have better initial focus

streamlined server file import and improved respective data usage calc

made internal options storage a lot more efficient

cleaned up a little cache code

export to zip improved to new file handling system

upload to repo improved to new file handling system

some db write priority stuff switched around and renamed

streamlined a little client db workflow

neatened client db exception handling

db updated message is now a popup

removed some orphaned methods in client db

client file and thumbnail server requests made a bit faster and neater

removed db loop from export and dump file fetch

neatened some client and server file 404 exception paths

further cleaned up the way file paths are calculated and fetched

further cleaned db in relation to files, and removed the old mime cache, for both server and client

removed orphaned db export infrastructure

add_thumbnails in client made simpler

improved http response parsing

refactored a bunch of http response processing

new exception display system added for caught exceptions

fixed a shutdown hang on certain delete orphans failure

version 80

manage tags and ratings dialogs have their initial focus corrected

custom filter now obeys tag parents for tag actions

fixed an annoying as hell multiple-page shared-thumbnail visual-selection-status bug

added e621 test

fixed e621 tag parsing, including new species namespace

might have fixed a missing thumbnail error

fixed system:hash

fixed some repo file downloading stuff

fixed some general db fetching file info stuff

made x files imported from y message a little slimmer

fixed minsize for the main gui frame

added better initialsize for the fullscreen frame

putting a thread url into the thread_id part of the dumper will auto-convert to the thread_id

added dumper's multipart form data generation test

added generic multipart form data generation test

fixed dumper unicode issue

error reporting in dumper is a little better

streamlined how mime is calced

massively improved how mp3, flac and ogg are parsed and validated

massively improved the internal import files workflow to use less memory and hdd reads and writes, particularly for video

similarly improved server-side import workflow

the way temp dir works is improved

sped up server file/thumbnail fetch

reworked how thumbnails are fetched in a couple places

started rework of how options is stored

wrote mp4 properties parser, but no mp4 just yet

version 79

popup messages will now report whenever a subscription or import folder successfully imports some files, with a button to show them in a new search

popup messages now wrap

slightly better error popup

many more errors reported through error popup

old logging system switched over to new messaging system

popup message manager will only show ten messages at once, now

selectfromlistofstrings now supports enter key to select

completely reworked how dialog cancel works; absolutely all dialogs should now close with escape key

reworked how a bunch of dialogs do ok

a huge amount of dialog refactoring

made ok button initial focus of all dialogs

couple of bugs in service options dialog fixed

rejiggered some button names and focus behaviour a bit more

some classname refactoring

muchly improved string->unicode handling

improved timestamp generation

fixed a clientserviceidentifier->text bug

reworked how namespace cache in tagsmanager object is calculated

improved instantiation of noneablespinctrls

fixed a bug in the thumbnails download daemon

changed the way daemons wait for the db, much to the better

moved daemons out of the db object

rejiggered writedaemon synchrony so exceptions work

fiddled around with some help links

default, non-maximised size of client is a little more comfortable

custom filter now has a popup that'll let you change the custom actions mid-filter

server now uses new synchronous logging system

fixed an options save bug for server

updated server diagram in help

added test for dialog selectfromlistofstrings

added test for dialogyesno

made a framework for testing that requires network stuff

made a newgrounds test

fixed newgrounds swf parsing

made a framework for testing that requires file reads and writes

fixed a graceful-exception bug in mime parsing

added test for synchronous import_folders

added test for delete import_folders

wrote a test for importfolders daemon

import folders no longer delete or reattempt failed imports; they'll just ignore them

import folders are deleted on update, since old objects are obselete

import folders won't try to do zips any more; they'll just ignore them

rejiggered how import folders does its path parsing to remove mime calc cpu usage

version 78

expanded parents testing with a namespace example

made getgstvcp in tags manager only return numbers for vcp

refactored ManageDialogs to their own file

refactored Exceptions to their own file

improved my testing framework so it can do wx gui elements

added test for dialogchoosenewservicemethod, fixed a typo

added test for dialogfinishfiltering

added test for dialogfinishratingfiltering

those two finish filter dialogs are a little simpler, now

set up a system so dialogs are a bit simpler, in terms of button event processing

added test for dialogfirststart

added 'add parent to tag' on tag right click menu

added 'add sibling to tag' on tag right click menu

made manage parents/siblings dialog focus selection more intelligent

'audio - any' mime search fixed

reworked audio embed container significantly

added embed button

all noisy mimes are now protected by embed button

you can now rescind pending file uploads or petitions from thumbnail right click menu

fixed a redundant bit of code in process content updates

in modify account dialog, set account expiry to 'does not expire' fixed

logging now happens a little more synchronously for client

popup messages prototype done

popup message for normal text

popup message for some errors

fixed a small redundancy bug in data cache

fixed a localisation bug when converting numbers (1234) to formatted version (1,234, 1.234, 1 234)

improved the code behind 'open selection in new page' in two ways

version 77

tag parents manager unit tests done

completely rewrote parents pair generation

rewrote parents pair retrieval to be much faster

parents manager is much more resistant to loops

in two ways!

loop detection fixed in manage parents dialog, as well

fixed how deleted parents are applied to combined service identifier

fixed how deleted parents are applied to specific service identifier

tag manager unit tests done, and everything was working ok!

shuffled objects around a bit, unified some tag code

tag manager merge unit tests done, and everything was working ok!

tag manager merge code simplified and result rewritten to a new, simpler class

small bug in how preview canvas fetches tags fixed

finished off import folders daemon

added import folders gui

added flac support

added ogg support

added a secondary layer of mime detection for greater accuracy

added a tertiary layer of mime detection for greater mp3 accuracy!

fixed an offset bug in mime detection

simplified and unified client/server file info parsing

import error handling improved

neatened how special thumbs are stored, refreshed, and retrieved

neatened how the bmp->png conversion happens

improved hash->mime caching for both client and server, to speed up http retrieval time for the new filetypes

better mime fetch in dumper, too

improved debugging messages for a bunch of my custom objects

version 76

made unit testing framework

tag siblings object now tested properly

hydrusdownloading functions now tested properly

added simple mp3 support to client, with external launch or embed

added search by mp3 mime

added mp3 to server

higher precedence deleted tag siblings now overrule lower existing pairs

tag sibling closed loop pairs ( a, a ) are skipped

neatened predicate sibling matching in two ways

improved tag autocomplete matching logic

skip logic improved for like ratings filter

skip logic improved and button added for numerical ratings filter

did a little work on reducing memory usage

reordered data cache's memory purge, which will make huge images load better

fixed a zoom bug with pdf buttons

added some select all, select none and refresh to thumbnail right click menus

reorganised menus a bit anyway

fullscreen tags on the left are now sibling-collapsed

version 75

fullscreenpopup window added

fullscreenpopup window can be dragged about

fullscreenpopupfilterinbox

fullscreenpopupfilterlike

fullscreenpopupfilternumerical

accuracy slider moved to popup

compare same image until done added to popup

keep on left/random/right added to popup

'don't ratings filter this' added to popup

that annoying as hell thumbnail selection drawing state bug is fixed

display of flash with only one frame fixed

downloaded tags now work again, sorry for the disruption!

deleted tags will now always show in manage tags dialog

fixed the various problems that were stopping DELETED->PENDING->CURRENT working for tags

fixed the children can only have one parent bug

fixed a deleted tag parents bug

a little counting logic improved in the special deleted_pending case

fixed a typo re uploading file to a repo

main guis statusbar now has a little 'db locked' indicator

version 74

flash scanbar added

made new mediacontainer class for media canvas

extracted scanbar to separate class

made animation frame tracking and control more sensible overall

previous_frame/next_frame shortcut now works for flash as well

fixed age predicate for > hours

fixed an important 'all known tags' A/C count cache bug when multiple services have the same tags for a file

if you have resolve_petitions permission for a tag service, tag siblings and parents petitions need no reason

slight change in how un-namespaced totals are calculated in A/C read

collection now obeys siblings

thumbnail upper text now obeys siblings

fullscreen text now obeys siblings

bit of siblings code cleanup

a little changeup to how showselectioninnewpage collects its media

/asp/, /gd/, /lgbt/, /vr/ and /wsg/ added to default dumper support

trying out making the server non-daemon, to see how that manages

version 73

every file operation from fullscreen was broke due to a single-character typo that got propagated

fixed local ratings, which were broken five different ways

ditched the deleted_pending concept, which was actually making things more complicated

possible fix to review services opening on bottom level

dumper no longer says it is still dumping on page close if it is done dumping

you can now scroll the dumper's upper comment box

moved site links over to github

fixed a typo re downloading a file that is already in the db and adding tags at the same time

local tags delete fixed

tag parents now apply globally, like siblings

thumbnail resizer daemon is less verbose and more helpful on IOErrors

fixed namespace colours in manage tags dialog

open selection in a new page now works for single-file selections

upload tags progress bar misalign fixed

namespaced counts no longer total up in A/C writes

manage siblings dialog now catches collisions when you put in the old, not the pair, and asks you what you want to do

age predicate now supports hours

while search is not synchronised, A/C will now query db when it might have queried the current media

think I fixed a sometimes-not-initially-showing-first-frame-of-gifs problem

downloaders now run parsed tags through siblings and parents managers

maaaybe fixed a cache counts error

wrote a little more sibling and parent help, and updated schema

version 72

remote tag siblings!

remote tag parents!

simplified update nomenclature

streamlined client-to-server update objects and processing

streamlined server-to-client update objects and processing

did a bunch of work on content_updates

did some work on service_updates, too

improved client-side petition handling

a bunch of changes to how uploadpending works

the way pending downloads are stored is improved

mappings tables are collapsed server-side

file tables are collapsed server-side

moved tag siblings and parents to content_updates

moved file upload/download to content_updates

reworked tag content_updates to be more streamlined

addfile/tagrepositoryupdate is made entirely content_updates, now

simplification in the way deleted content is stored

repo update processing string is muchly improved

repo update processing is smoother!

a slight change to default behind-the-scenes searches

improved how the server creates updates on service init

improvements to service_identifiers_to_statuses_to_tags sub-init and related parts of tags_manager

some misc nomenclature improvements

fixed a denypetition bug for file repositories

fixed optimised petition handling in server

fixed posting tags in dumper

fixed a problem with getunknownaccount

server now analyses on vacuum, like client

fixed a lolbug in thumbnail resizer daemon that was continually resizing thumbs

fixed a bug in cache counts, which should fix the 'pending menu disappeared' problem

corrected num_archive count in file repos

downloading files is simpler and faster; download menu is simpler

fixed server initialisation

updated to new sqlite version

version 71

collapsed the four mappings tables into two tables

merged the two active_mappings tables into the mappings table

made a great number of changes to how mappings and active_mappings are stored and processed throughout

'active' and 'null' nomenclature is now 'combined'; null service_ids are now just ints

improved deletepending so it isn't so tough on the a/c cache

tags regex dialog entries 'for all files' and 'just for this file' was all broke

A/C read now bumps the exact match of the entry to the top of the list, if its count is non-zero

fixed A/C for weird-character queries, like '['

the dumper now makes success/error noises as appropriate

you can turn these noises off in the new sound tab in file->options

screwed around with garbage collection while checking mimetypes during pre-import

adding a tag parent will spam-add the actual parent tags to every child instance for the appropriate service

updated db diagrams

revised my sibling chain collapsing algorithm

locked db on init dialog message is improved a little

system:limit added to combined file service searches

num pending menu counts now split into with (pending/petitioned)

corrected a server db index oversight

newgrounds title tag fixed

version 70

tag parents db stuff

tag parents manager

tag parents predicate and matches indented expansion

and drawing it!

tag parents top result reordering

tag parents top result insertion

tag parents actually doing what they do

polished off tag parent help

made a full-search table to speed up tag A/C requests

added tag full-text search

reworked how tag and file services add and reset, to reduce A/C time

reworked recalc active mappings to be more beautiful, if not faster

fade animation timer improved in several ways

thumbnail fetch made much smoother

newgrounds artist downloading for games and movies added

newgrounds subs added

download panel input now highlights on init

4chan filename tag added

the setfocus on filter close is neater

age phrase is now 'imported [time] ago'

age is now shown in fullscreen

can now copy file from fullscreen, from menu or shortcut

4chan pass authentication improved

fixed a couple tag service precedence sync bugs

version 69

first dialog for tag siblings

tag siblings db table

tag siblings manager in controller

tag siblings display in tagsboxcpp

tag siblings display in tagsboxflat

tag siblings display in tagsboxmanage

tag siblings display in tagsboxactiveonly, via A/C dropdowns, for both read and write

top result sibling switcheroo in A/C write

A/C db fetch now does siblings too

db tag search does siblings

siblings help, with some nice mini-charts

I fixed collections, which were typo-broke; I'll make sure it doesn't happen again

more granular subs error handling, meaning individual file failures won't crash an entire sub

individual subs can now be paused

e621 fixed for real this time

A/C improvement that slows tag updates a little but should stop A/C lag after an update

as a result, trying out dropping the CPU-intensive fatten_ac_cache maintenance call

A/C read will now update system preds every time you click on it, so inbox/archive counts will stay accurate

A/C "all known files + tags" will no longer show the mega-laggy total file count

a list -> tuple convenience fix in sanelistctrl

if you don't have any pixiv credentials set up, you will now no longer get the option to start downloading pixiv stuff

fixed a tiny typo in the thumbnail resizer that made it wait far more politely than was intended

slight change to ratio system pred that fixes some lockups, sometimes, I think

fullscreen flash and video will get a pixel of whitespace on the right

version 68

fullscreen view now takes addmediaresult

export to zip!

import from zip, with tag regex stuff too!

export to encrypted zip!

import from encrypted zip!

import encryption help in advanced.html

import process code processes mimes less jankily

delete after successful import checkbox added

import regex tags dialog now has # namespace

import regex tags dialog collects its info a bit more intelligently

new confirm exit client option

subs will save their progress every 20 files downloaded, so pause/restart is less punishing

simplified thumbnail cache retrieval

some thumbnail waterfall and animation tuning, let me know what you think

e621 fixed

danbooru fixed

reworked booru gallery page num calc, which was completely screwed

DA fixed for most cases

booru thumb parsing made a _little_ more permissive

server now uses the same new file storage system as the client

download cancel buttons have a little better feel

new thumbnail prefetch thread makes thumbs load a load faster

fixed a little rubbish naming semantics in ClientGUI

reorganised and streamlined much of the encryption code

updated a bunch of help screenshots

version 67

subscription db access improved

subscription 'delete subs seemingly at random' bug fixed

some nice subs help

subs and repos, if changed during a pause, will restart with new changes

subs and repos will automatically pause while their respective dialogs are open

new namespace | regex listctrl in regex dialog, instead of old sctvcp rubbish

new /aa/aa...0 file storage system for client

fixed deleteorphans mime issue

export and copy files now export writeable files, not read-only

rejiggered daemon db access, improving maintenance reliablity

dumper and 'show in new page' pages now process content updates correctly

version 66

subscriptions done! works for all normal download types

pause repo sync, pause subs sync

all download sites moved to the new system

lots of small changes to how download code works

pixiv tags is renamed to pixiv tag, since it only does one!

fixed DA

fixed DA again!

fixed DA tag parsing for all the diff types of username re http://help.deviantart.com/106/

fixed pixiv, somewhat

fixed giphy

fixed boorus for gallery_advance_num > 1

boorus reset in db to default

fixed copy files

fixed getmime

a silly parent assignment for A/C meant they were not closing with pages

made the cpu burn for the thumbnail resizer a bit more polite on hdd

repo sync daemon and subs sync daemon combined

some timing adjustments in sync daemon

subscription_type constant -> site_download_type

fixed temp folder being cleared on startup (a read-only issue)

advancedoptions classes now support setinfo

version 65

added subscriptions dialog

added prototype subscription daemon, but not yet activated it

added downloader classes

began reorganisation of nearly all download code

moved parsing around

moved advancedhttpconnection around

moved serviceupdate around

moved some content_update stuff around

moved some constants around

files in client_files now have extensions

files in client_files are now read only

pdf launch is simplified as a result of this new ext stuff

fixed find similar images; it was just a typo

also fixed initial predicates string display

rejiggered the thumbnail resizer again; due to my stupidity, it was causing lag

thumbnail resizer burns a little more cpu when there are >10,000 or >100,000 thumbnails to render

it also now does thumbs in random order, for a couple of good reasons

upgraded sqlite, hopefully some queries will run faster?

upgraded to python 2.7.4. please report any weird errors

version 64

got rid of system:not_uploaded_to - now system:file service, which is a lot more powerful

some display changes to system predicates to make them a bit more human-readable

added export dialog

made a couple small changes to help about the export dialog

collect by rating

sort by rating now works for collections, using an estimate (only accurate when also collecting by that rating)

built a custom radiobox class and added it to ratings dialog so it has radio buttons again

new ratings filter for local_ratings_like services. works just like normal inbox filter

added a little info in help about this new filter

new session manager for pixiv and hentai foundry, reducing number of session inits the client has to do

new init welcoming window for first boot

new politer error message for when db is locked on boot

shift+arrow keys now pan in fullscreen mode. you can change them in options as normal, if you like

resize thumbnails daemon is now part of maintenance thread stuff. hopefully less blocking now

inbox pages will remove media when you archive stuff, again. this is just a bit I forgot to update from last week's predicate overhaul

fixed and changed the way hitting the change tag service button works

I fixed previews and fullscreen cache estimate in local options too!

fixed giphy so it works with their new api

rejiggered the update notification order to be a bit more accurate

updated auto-setup code and help to use new no-ip domain rather than old raw ip

v64 will also update the old ip to the new no-ip domain automatically

bit of variable renaming

cleaned up a couple menu memory leaks

updated a little of future.html

version 63

added include/exclude namespace predicate

added intelligent delay to A/C to smooth out db requests

rejiggered the A/C code so that spamming/pasting several characters and then deleting some will re-search for matches as appropriate

cut size down to just B, KB, MB, GB

maybe I fixed system:ratio in certain cases!

some general predicates display syntax changes

(archive/inbox) and (local/not local) mutual exclusivity with new system

system:num_words

right click -> open selection in new page

upgraded from PIL to Pillow, to zero noticeable effect lol

had another look at dodgy animated gif transparency; couldn't figure it out

CMYK jpegs images now supported! interlaced pngs simply not supported with PIL/Pillow yet

numpad delete added as a valid shortcut key

removed an old buggy deleteorphans line

new thumbnail resize daemon to fill out the thumbnails directory in the background

collections with only one file will now be converted to singletons

screwed around with locale settings. not sure if this will break things!

manage tags and ratings now in fullscreen rmb menu

on downloads, ( url -> hash ) pairs are stored in db, even if the file was already deleted or already in db

a few additions to help

reworked the spinctrls system preds to not have 100 as default max

changed a bunch more spinctrls, including memory sizes, in local options

and a statictext that wasn't showing number of thumbs for cache size

bugfix in ratings filter

version 62

remade a/c dropdown to a better, less buggy class

reworked a little of the a/c dropdown focus logic

reworked a little of the a/c dropdown show/hide logic

a 'sometimes hangs on "gui" on startup after crash' bug is fixed

can copy from listboxes with new right click menu

dumper page will protest if you try to close it while it is still dumping

import page won't protest if you try to close it while paused

num_words will show on pdf right click

fixed the network version mismatch exception, exactly one week late lol

couple of tiny menu memory leaks removed

couple of misc gui tidies

custom filter favourites save and delete buttons will grey out as appropriate

bit of general code cleanup

we nested classes now

moved contentupdate class from cc to hc, in prep for repo update rewrite and contentupdate gen

version 61

session db tables

session manager objects

session net code done

session failure recovery

registration key table

registration key requests

some registration help added

harmonisation of many http request names

extracted uploadpending code from db transaction, making it a lot more polite and architecturally correct

pixiv now parses the ascii creator name as well as the japanese counterpart. No option to choose between them yet!

improved processcontentupdate to support more actions, making db a little more serial

home and end now scroll to focussed media

some http timeout tweaks, to stop the 'can't connect to server' problems on complicated queries

a typo in modify account types was fixed

manage tags dialog shouldn't fritz out quite so much when a service doesn't have post_data permisisons

version 60

pixiv account management dialog

pixiv gallery parse

pixiv page tag parse, including creator and title

pixiv user download

pixiv tag download

collapsed the page chooser a little to fit pixiv in comfortably

pdf support!

pdf launching via preview/fullscreen button

pdf header parsing with num_words rough estimate

system:mime=application/pdf

system:mime=application

custom filter favourites done! with default, previous, save, save as and delete

some small changes to temp folder logic

vastly improved scroll-thumbnail-prefetch logic. important bits all rewritten and made sane. data and gui rows line up better, with fewer bugs

vastly improved thumbnail fade. now attempts to render at 60fps, and calculates alpha much more efficiently

thumbnails won't fade if they are off-screen; they'll just draw in one frame

fixed thumbnail last row drawing problem (again!)

due to daemon-db-spam, db maintenance check was only firing on pc wake from sleep, now it fires properly every twenty idle mins

maintenance timer won't fire directly after waking from a sleep

deleteorphans copes with alien files a bit more gracefully

A/C appearing on top is mostly fixed

some general A/C weirdness is fixed/made better

reworked some of the thumbnail selection and scrolling code to line up better

thumbnail window will stop spamming 'scroll to focus' every time something small changes. it should only do it on fullscreen close and non-ctrl or -shift mouse/key events now

some general focus code was looked at

ratings filter now supports the fullscreen_switch shortcut

just a couple changes to help to include the new 'f' fullscreen_switch shortcut

10s timeout on http queries

http redirect code is improved

http cookie handling code is improved

made hf session establishment a bit cleverer

version 59

new vastly improved 'collect by' dropdown

fullscreen 'f' shortcut

fullscreen default option

added tumblr prototype parser, with tags

server init and repository init typos are fixed

a misaligned sizer was fixed

a bunch of spinctrls no longer init at <= 100

the problem with the last thumbnail rows sometimes not drawing properly should be fixed

A/C lag calculates a little more intelligently

manage ratings for collections now works

version 58

new custom staticbox sizer/panel put in about fifty places

big layout changes in management panels

I made a new, better set of sizer flags for nested sizers

testing out several new borders

several other small layout changes, lining things up

a few colour changes with the new classes

advanced options class is better, more streamlined

advanced options class is less flickery, I think

A/C spazzes out less when dragged about

A/C queries db less annoyingly, especially when you type fast

the thumbnail display draws its whitespace a little better, filling in the right-hand and bottom-end gaps

importing thumbnails now add to the last row one thumb at a time (rather than at one row at a time) once the scrollbar appears

shift + f7 now sends things back to inbox, like a reverse archive.

this works from thumbnail menu

and in fullscreen

new menu item 'remove these' for thumbnails, anywhere, including dumper!

'tags just for this file' now works a looooot better! it unions all selected tags, and detects deselections better

giphy downloader added

giphy tags added, thanks to @fluffy_cub

copying file now _also_ copies paths to clipboard; depending on where you paste, you'll get the files or the text appropriately

new shortcut (default ctrl+m) for 'focus media'

focus search now works for manage tags and tags regex dialogs

think I worked out the last of the up/down arrow cursor shortcut problems

a ton more code cleaning

hentai foundry artist not found 404 error detected

in regex dialog, tags are cleaned before display on screen

4chan ban is now recognised, and thread dumps are hence suspended

when ordering tags by incidence, disputes are now resolved by lexicographic

fullscreen init is a little faster for the first image

select imageboard dialog now toggles tree with double click

select imageboard dialog is also taller

version 57

added quick vs accurate slider for numerical ratings filter

new slider will remember where you left it last

ratings filter internal comparison logic is more sane

'already rated' will now show the specific rating beside it

ratings filter (hopefully!) chooses better comparisons, in about four different and new ways

you can now ratings filter just one file

ratings filter code is neatened a whole lot, especially on my horrible variable names

I may have fixed an equality ratings filter bug

internal rating data -> pretty text for display is more flexible and accurate

going back several steps on ratings filter will no longer forget to re-rate already rated files

delete from dumper no longer screws with the indices

dumper is more stable, less likely to bail on the whole dump if something odd happens

dumper should recognise duplicate file errors and recover the fail

if the user has idled for the last twenty minutes, the db checks to see whether it can optimise itself

fixed flickering on fullscreen drag

fixed flickering onn fullscreen image init, meaning smoother mousewheel/whatever scrolling

fixed a graphical glitch in manage boorus dialog, and a bunch of others (removed all staticboxsizers)

fixed a similar glitch in manage ratings dialog - it now has dropdowns; I'll probably make it radio buttons again soon

fullscreen page navigation (previous, next, first, last) shortcuts are now customisable

custom filter now supports these new page nav shortcuts

can now do home/end and page up/down in shortcuts

shortcuts listctrls now sort by action by default

I improved the my custom listctrl's code generally

little less ugly code, here and there

fixed a whole ton of regular little bugs

version 56

new existing tag sort dropdown

an option to set default existing tag sort

dumper error logic improved

dumper now does random sort properly!

double buffering hilariousness

some reordering of the local options dialog

made a 'shortcut' entry object

made a parent frame object

collection thumbnails will redraw if the sort changes

add media and add thumbnail data and graphics logic is improved

4chan pass now authenticates for an entire year

bit more in help about the autocomplete tag entry

tag petition processing speed is a loooot faster

version 55

custom filter prototype ready!

added some info on custom filter to help (in advanced.html)

you can now set a default tag service in options

tag rexeg dialog's listctrl refreshes without scrolling up to the top every time

tag for this file in tag regex dialog applies to all selected, not just the first

scrolling in autocomplete entry will scroll services in manage tags and tags regex dialog

ctrl+scrolling in autocomplete entry will scroll through results

complete rewrite of image prefetch calculation

couple of content-update display bugs in ratings filter fixed

in manage ratings dialog, sliders will preset to best guess at current score

dumper should be fixed

resized active predicates box (much smaller)

a deprecated call updated

added some forum extensions

version 54

several new flash key and mouse event bugs fixed - now: mouse in = to flash | mouse out = to frame

the problem with scrollable management windows not drawing things on scroll is fixed; staticboxsizers now replaced with "- title -"

ratings filter now accepts several shortcut keys, directing to whichever window the mouse is over

the ratings filter windows will redraw themselves appropriately on content and service updates, e.g. changing tags with F3

ratings filter will choose the files to compare to more intelligently, both from the db and the currently-rated pool

the way services are identified behind the scenes is entirely overhauled and improved

the review service dialog is better as a result

all dialogs will now cancel-exit on pressing escape

started fade in and out for thumbnails

up cursor arrow in filter now skips

the tags regex dialog is refitted so it will refit on small screens better

dumper now dumps in its own thread (no longer locks up the gui while uploading)

the calculation of num_frames for flvs was off by a thousand, lol

synchro/wait shortcut typo fixed

inbox and other icons moved to background in fullscreen

fullscreen makes less db queries r.e. ratings

a couple managementpanel resize issues corrected

a couple drawing/flickering issues corrected

some updates to help

I made a forum to my site, and added links in the client and help

version 53

several terrible ratings filter bugs fixed

ratings filter can now rate files internally

ratings filter frame now has a statusbar

system:rating:service=uncertain added

upgraded to wx2.9, which included a ton of bug fixes

sort by namespace wasn't working due to an odd typo

collect by logic changed a little so that 'none' values are collected

the shutdown lag from v52 is corrected

version 52

ratings filter prototype is done

bit of help docs

more intelligent cpu burn on shutdown

dumper can manage flood time error, and recover from it

whole load of work done on mappings optimisation, meaning mappings updates, uploading mappings and approving petitions work about 100x faster!

forgetting pending uploads now throws up an ok/cancel dialog

had another look at overwriting deleted tags

version 51

dumper deals with failed uploads better, although there is more work to do.

flv support for client and file repo!

reworked tag services' permissions calculations on thumbnail menu

new fatten A/C cache on shutdown

system:untagged works again for null tag repo ('all known tags')

advanced import options now default no limit for size and resolution

ratings service now have a little info on the review services frame

new shortcut keys (ctrl-b and -n) for animated gif previous/next frame in fullscreen

added thoughts on plurals to tag schema help page

hentai foundry downloads now interleave pictures and scraps

removed db locks on file/thumbnail reads

you can open the page picker frame with the mouse by going view->pick a new page

the copy->whatever thumbnail menu is more concise

system:hash now works for null file repos ('all known files')

there is now a title: regex on the tag regex dialog

servers now store updates in a folder, not inside their dbs.

couple misc changes to help pages

version 50

4chan pass added

captcha timings in dump are better

dumper's file ordering is less insane

some general updates to regex tag dialog

new box on regex tag dialog for tags for single files

regex tag dialog now supports arrow key up and down to change repo

entirely new and vastly improved listctrl class

collect by is now a thousand times more complicated, but also much more powerful

the top-left media-flicker display bug in fullscreen is fixed

some logging improvements

some tag cleaning improvements

thumbnails have less cluttered tag headers

some small changes to help

fullscreen precache doesn't do gifs any more, which should make things smoother

got rid of the awful view pending uploads dialog. I'll improve it later

started remote ratings, but its not done yet

version 49

local rating like service

local rating numerical service

ratings dialog

ratings drawn on preview and fullscreen canvas

search by rating

system:rating in options

sort by rating

a ratings help page added

custom shortcuts

overhauled shortcuts code

a new, comprehensive regex menu in the regex tag dialog

extensive new hentai foundry search and sort options

a new option regarding the search style of tag A/C entries

thu dumper can now dump with tags

regex tag dialog selects tag A/C on page change

regex tag dialog's button now now shows in import dialog even if there are no remote tag dialogs with POST privilege

A/C is slightly faster in all cases

preview canvas now updates on a content update

numtags = 0 and numtags > 0 now work for pending tags

shift- or Ctrl-Click no longer deselects

listbook renaming and resizing bug fixed

collections thumbnail is fixed

sometimes, an empty tag, '', could be entered into pending and screw a lot up

sometimes, a null tag could be entered into pending and screw a lot up

advanced tag options has better checkbox defaults

some help screenshots updated

the db's processing is slightly rebalanced

the db's startup is slightly rebalanced

a misc gui bug was fixed

a misc gui typo was fixed

version 48

New copy path and copy url menu entries on thumbnail right-click menu

Gif scrollbar

Hentai Foundry parsing errors fixed

Hentai Foundry 404 errors fixed

All the html parsers are rewritten using BeautifulSoap

Flash now zooms full, but not fit

Filter is now F12

Help is updated in a number of ways

Up/Down on empty input to change repository in manage tags dialog is fixed

The servers now support OPTIONS requests

Several status typos fixed

I started local ratings services

version 47

Hentai Foundry interface is done, for both artist and tag searches

A/C gui reorganisation

A/C file and repo buttons added

A/C cache count alterations to match those buttons

A/C tag processing is cleaned up

Search and thumbnail-action improved to support null file service or non-null tag service

system:numtags=0 and system:numtags>0 optimised to be much faster

Ctrl+I now switches searching immediately/waiting

z now switches between zoom full and zoom fit in fullscreen view

advanced import options now has exclude already deleted files checkbox

Thumbnails choose the way they access the db more intelligently

ManagementPanel colour is changed, hmmm

Some DB cleanup

Some general code cleanup

Some download logic improved, including timer

Some download status improved

Download 404 notices better

DeviantArt now initially shows 'artist username' in its searchbox

Some code nomenclature improved

I've made a changelog.html, compling all changes together.

version 46

DA parsing CLEAR

rule34@paheal parsing CLEAR

tbib (another booru) parsing CLEAR

Canvas rearranging

Preview now shows tags

The autocomplete tag write dialog will promote existing exact matches to the top of the list, even if it is not the most popular.

Autocomplete lists now scroll with the mouse.

A/C scrolling is a little more accurate.

Pressing up or down on an empty A/C entry will change tag repository.

A sledgehammer has been applied to the python garbage collector.

The code that navigates boorus is better.

The import queuing/pausing/cancelling system is better.

Booru tag-processing is better.

The import-queue gui is slightly better sized on small screens.

The new local tags service now shows up in advanced tag options.

Even more flicker is removed from the main thumbnail view, this time when clicking on whitespace. (it was a stupid bug)

Serverside db-backup now includes the WAL file.

Manually adding an admin service is fixed.

version 45

Local tags: DONE

Thread watcher: DONE

Fullscreen now shows tags

Preview shows some more file + tag info

Listbooks load their panels on demand.

Local options dialog is now a listbook.

Some incorrect �sample image� downloads from danbooru are fixed. Danbooru�s inner workings remain a 404-cluttered mess though.

Some remaining utf-8 html parse errors (where a limited content-type header is given) are cleared up.

Another connection problem (a certain case of the server closing the connection) is now properly recovered from.

Some flicker in the thumbnail view is reduced, but my slow laptop still shows some, so I need to revisit this.

Autocomplete tag boxes now have little tooltips better describing their search domains.

version 44

All database triggers are removed and replaced with more efficient inline code.

Cached number of files, namespaces and tags for a tag repo are kept updated, so review services should rarely take very long to load.

Cached number of A/C counts (current and pending) are kept updated, so A/C results should rarely take very long to load.

The way A/C cache searches are done is improved.

The way A/C cache entries are calculated is vastly improved.

The way several db SELECT DISTINCT * and SELECT COUNT( DISTINCT * ) queries are performed is improved.

The custom httpconnection object is improved to better deal with server-side connection closes. (Cannot connect to server problem)

The custom httpconnection object does http redirections more intelligently.

The custom httpconnection object now parses non-ascii content-type (e.g. utf-8) properly, according to server-side header. The tag �pok�mon� will now parse correctly.

e621 and rule34@booru.org are fixed. (just a couple parsing-variable changes)

If some tag parsing entries are checked on a booru download page, the tags will be downloaded, parsed and applied even to files already in the database.

The mouse cursor will remain visible in the fullscreen browser while the right click menu is open.

I corrected some which->that grammar in a couple dialogs.

Auto-repo-setup (from the help menu) was broken in a very stupid way, but I fixed it. I apologise for not noticing.

Thumbnails now show creator and title up top. They are a bit crushed, so I may rearrange them.

version 43

Updated import + booru code

New booru buttons

Cut tag parse bug fixed

Download preview instead of real file bug fixed (I think)

Fixed the �thumbnails not appearing on import� bug

Import flickers less

Media Classes rewrite is done, although I�m not totally happy. Have to think about it.

A problem with pending tags applying over existing tags no flushing correctly is fixed.

A/C cache is better, but still a little slow on slow computers.

system:duration=0 is fixed.

version 42

Autocomplete tag cache DONE. Please tell me if you encounter excessive slowdown or incorrect counts.

Autocomplete tag logic improved in about five ways. Tags should appear more intuitively and with more accurate counts.

Copy and paste buttons in manage tag dialog to facilitate spamming many tags from one file to another.

Imageboard dump is fixed!

URL download got fixed last week and I forgot to say!

A peculiar pseudo-foreign-key bug with the active_pending_mappings table is sorted. (leading to more accurate queries and counts)

Since display of large queries is so much faster, the �woah! 12,000 query results� warning is gone.

Fullscreen precache timings are rebalanced, hopefully for the better!

The importing gui classes are a little smoother, and thumbs should load one by one, not row by row, once the page height is exceeded. The scrollbars should be less janky.

Page transitions are smoother.

Importing from URL will now load those urls that are already inside the db (hella fast, too), just like importing from hdd.

The double-update-download bug is finally fixed! Now you�ll download update number 258 only once, and not twice in a row. (I hope!) It never mattered much, beyond a little CPU, but it annoyed the hell out of me. Working out what was wrong took ages. (Long story short: Python�s Priority Queues are not parallel FIFOs!)

There was a weird bug in pending tags sometimes going to higher tag-precedence services, rather than the one chosen in the manage tags dialog.

There was a typo in reset service.

More content update unification in the db, making things much neater behind the scenes.

I found a potential hang in the db-shutdown code.

I found another potential hang in the db-shutdown code!

The number of messages in your inbox is now displayed on the statusbar.

Bold, Italic and Underline now work with Ctrl+B/I/U in the message compose panel.

The �don�t receive messages from Anonymous� checkbox in Add, Remove and Edit Services for message depots now does something.

system:unread in message search now has a count.

A draft will remember its saved changes when you click on another conversation and back again.

You can now save a draft with zero recipients.

version 41

Rich Text Control with Toolbar in messaging.

Message unread/read status now syncs to message depot.

Can set defaults for file system predicates.

Animated Text class written.

system:unread tag in messaging.

Import bug fixed.

Messaging search logic improved.

Message depot default check time down to 180, and daemon thread improved to manage with that.

Application startup is more stable.

Some AddMessage logic improved.

Drag+Drop service fixed and improved.

Tags added while importing will now turn up in autocomplete.

I unified more of the client�s content-update db calls. Just making stuff neater.

I massively simplified the autocomplete tag code in preparation for another bash at doing autocomplete cache. A/C is far too slow atm, imo.

Some A/C counts are still off, particularly when there are a mix of current and pending. I couldn�t figure out what is causing it, but I�ll look at it more as I do the cache.

In add, remove and edit serivces, you can now specify if you want to receive messages from Anonymous or not. Unfortunately, because I forgot to implement it in DoMessageQuery, this checkbox doesn�t actually do anything yet! Whoops!

version 40

Inbox/Archive for messaging works. The inbox icon is back, and F7 to archive as usual.

Unread/Read for messaging works. Click on your status next to a message panel to change.

All message participants are shown in the conversation listctrl.

The message page will open with �system:inbox� active, ordered by newest message.

I fixed conversation listctrl sorting in a couple ways.

Some bugs in the db->gui update reporting system are fixed, so the compose window and draft panel will close properly on a message-send, and the conversation listctrl will swiftly update unread counts and inbox status and so on.

The draft panel�s save button wasn�t updating sometimes when it should have been.

html links in messages will show in the main frame�s statusbar and launch in your browser when clicked.

I reworked how many system tags are searched in file queries to make them muuuuch faster. Now, any time you go system:hash, :size, :mime, :age, :width, :height, :duration, it (should) be significantly faster than before.

Ever have a png file that had an ok thumbnail but a black preview/fullscreen? It was probably an �LA� image, or �greyscale with an alpha channel�. Display of LA images is now fixed.

Streamlined the way file �archive� and �delete� commands are sent throughout the application, with an eye to eventually adding all other user-commands to the same system (to make it simpler, and also to do undo-redo).

I�ve updated the help docs, including the section on how messaging actually works.

You can now enter system tags with enter/return from the keyboard, not just double-clicking with the mouse.

version 39

A whole bunch of stuff I didn�t write down that generally made messages work beyond sending �Hello World�.

Unicode messages now work.

Newlines in messages now display properly.

Fixed contact saving and drag-and-drop to the manage contacts dialog.

You can now add a contact using a contact address (assuming their service is v39).

A bug in contact renaming is fixed.

My contact (hydrus admin) is now added automatically.

Messages are orphan-deleted properly server-side.

Message depots you control now get an entry under the admin menu.

The conversation listctrl is now properly sorted.

The HtmlWindow that shows message body is now resized to its minimum needed height via magic that I ain�t gotta explain.

Messages that fail to send can be retried by clicking on the failed.

Some draft save/send button enable/disable logic was updated.

I switched around some server-side account logic.

You can�t zoom a flash window bigger than the screen.

I�ve made the mouse appear in fullscreen when you move it, and this includes while you drag. See how you feel about it. I�ll make options to change behaviour if people care.

Better zip structure. (No more CWD rubbish)

New tar.gz release with only the code and help. Code is 1.2MB, help is 14MB lol. I�ll improve the structure of this release next week. I threw it together by hand today, but I�ll improve its structure next week.

version 38

messaging v1.0

version 37

messaging prototype

version 36

animated gif fixes

new page chooser

svcp->cstvcp in fullscreen too

new import progress

just start the server on this computer, please

I think I have scrolling to the current selection after an archive fixed now.

Started an experimental simpler way to send commands to the db, which may open the way for undo-redo.

Capitalisation option in file->options->gui. It only does the menubar for now, but I will expand it.

Custom sorts/collects. Under file->options. Check it out, play with it if you care about it.

Some program initialisation error handling is cleaned up.

Can now no longer start a server if the admin port it wants to use is occupied. (can�t run two servers at once on the same port)

I broke dump with a typo in a previous release, I think! It is fixed now.

version 35

Slight speedup of review services frame loadtime.

SVCP now replaced with SCTVCP and CSTVCP for now, with more planned next week.

Repo sync daemon adds db jobs less manically, now, locking up the gui less.

Repo sync daemon reports successful updates to the log.

Tag petition review page has a visual bug fixed.

ThumbnailMediaPanel manages its underlying bmp canvas more intelligently.

Some ugly page-closing code was cleaned up.

There is a �regex examples� popout window on the regex-filename-tag import dialog, with copy-pastable examples of python regexes that�ll work for common filename-parsing situations.

Home and End keys now select the first/last thumbnail as well as scrolling.

version 34

DB consolidation, and foreign keys and triggers with it. The db is less prone to orphan ids and miscounts.

For long-time users, all orphan ids should have all been deleted, reducing weirdness.

Total size of local service is now tracked accurately.

Archive while importing should work a bit better now.

Update logic should be more reliable, as should all service_updates.

Archive count for file repos should be less wrong.

Fixed two (!) typos in system:age predicate code.

File upload is fixed (not sure if it was ever broken in release, but I fixed the stupid thing I did anyway)

DB (and server) errors should report better now, revealing inner-db code in the traceback. (and printing it to the appropriate log)

Improved the mime_to_enum object for image/jpeg, image/jpg, image/jpe, image/wtfaretheygoingtocalljpegnow.

HTTP connections can deal with some esoteric commands, like 301, without throwing a fit. All http goes through the same code.

Danbooru should now be in your booru list, and it _should_ work.

Pausing/Cancelling a booru gallery parse should now work.

4chan dumper now understands captchas being wrong and the thread not existing; in the former case it�ll let you put in a new captcha and try again, in the latter it�ll stop the whole dump.

Some general code neatness improvements.

version 33

Boorus are working for users who started last week

The server can now manage many more simultaneous requests without bugging out.

The early freeze/import freeze due to simultaneous access in the client is fixed.

The client�s file+thumbnail server (details in help�s advanced.html) works again.

Client DB now stores md5 and sha1 for all local files.

Client DB has a larger cache size.

The freeze caused when importing thousands of files at once with tags is fixed.

The yaml imageboard import (when you drag and drop a .yaml file onto the imageboards dialog) now adds form fields correctly! If you had problems here, check out install_path/static/default_imageboards.yaml to reset them!

The autocomplete counts should be more accurate, particularly when there are a mix of current and pending tags in the db.

The autocomplete in the regex-path dialog will now search the db correctly (it was misconfigured).

Autocomplete is also just a _little_ faster

File imports are now low priority, so gui activity should still happen nice and quick even while an import is occurring.

The review services frame will now show how long until the next update will be checked (see picture above). In case of error, it will show how long until it tries again.

The review services panel and manage services panel are of a better default size, and they use my new listbook class exclusively, rather than the old notebook.

I�ve added some tooltips on not-obvious widgets.

Content Updates have vastly improved processing speed on a thumbnail view (x100 in some cases!). Tag updates particularly will not be so slow to process if you have a large search open.

The import gauge will now show the correct range according to the current queue size, not always the default 100.

Active Import Pages will throw up a yes/no dialog if you try to close them.

Some beautifying of code syntax. �\n� is now os.linesep, which means exported stuff should show up better in notepad.

Some changes in network protocol, although now I think of them, I realise I forgot to update the network version. They aren�t important for most users, so w/e.

version 32

client db serialisation DONE

Cleaned up a dirty-bmp canvas init issue.

Resized some dlg stuff that was too tall

Cleaned up a bunch of unrelated db code

Tightened the pubsub system so it integrates better with wx.

version 31

booru prototype

Fixed some problem with last error timestamp

Fixed a bug in repo bandwidth tracking

Fixed URL registration with the client db.

Made downloading URLs asynchronous again

Made repo synchronisations not block so much, and not for nearly as long. You should be able to do queries and click on thumbs (albeit at a slower rate) while the client is downloading and processing updates.

version 30

Thumbs are exported, and, along with perceptual hashes, now treated as a global pool, reducing downloads and speeding queries.

Thumbnail resize is now asynchronous, done as each thumbnail is requested. Changing the thumbnail size in the options dialog happens real quick, and if you don�t like a particular size, you can change it back just as fast.

Temporary DB locking mechanisms ensure serialisation, less inefficient locking.

ImportPage superclass works a lot better, in a lot less code. DB import code is now even simpler, and much faster, especially for redundant files. The new import page will also tell you how many files were successful/failed/already deleted/already in the db.

Fetching images for fullscreen or preview is fully asynchronous, and will no longer block the gui thread.

Some query logic is improved.

You can now import and export sites/imageboards from the manage imageboards dialog. It�ll export to a .yaml file, which is easy to understand with a decent text editor. Just drag and drop onto the dialog to import.

Selecting thumbs has a little less preview-flicker now.

The new tagboxes now render numbers prettily, so 4123 -> 4,123.

(+) namespace:tag now colours properly.

The background for selected tags is now an appropriate colour.

You can now choose the colours of any namespace you like in the normal options dialog.

Clicking in whitespace in the new tagbox deselects, rather than select( 0 )s.

Spacebar, PageDown, Return and all the other special command characters are now propagated to flash in fullscreen view, just as long as the mouse is over the canvas.

version 29

Big changes to the db�s file storage, as above.

database->vacuum is reinstated, database->reindex is retired. I recommend you run a vacuum once you are updated; it�ll only take a minute now.

Rollback and initialisation bugs are fixed.

There�s a splash window now that can offer client-boot feedback.

The way scrolling-to-focus is calculated is improved.

Pretty much everywhere you see tags will now show coloured tags! I finished up my own tagbox class, which for now will show tags coloured in the default booru scheme. It also loads in a few milliseconds, rather than the ~0.5s of the default listbox. It scrolls faster, too. I�ll have user-customisable colours next release.

Files with invalid resolution, size and mime are now filtered out before dumping to an imageboard.

A number of misc other bugs were fixed.

version 28

I fixed the �searching immediately� bug. It was a stupid typo as usual.

Post-Filter scrolling should be a bit more reliable.

Some more log events.

I did a lot of work on my new tags box, although the actualy gui elements will have to be finished for v29.

Manage Imageboards wasn�t saving properly. It is now.

My new progressdialog happens in several places. It is a little neater than the one in wx; just better for my purposes.

I reworked some of the transaction stuff in the db so that I can atomically queue pubsub events. (this means file import are less juddery)

After becoming very annoyed with wx�s listbook/labelbook/whateverbook, I wrote my own. It�s all over the place in my dialogs now. It is tighter, more reliable, and can scroll.

I reworked some of the way sets interacted.

There was an infinite recursion bug sometimes, when uploading files to a file repo.

version 27

Cleared out some old unused library imports, so the client is slightly smaller.

Had a cyclical import (!). Luckily, python is forgiving, but I have still fixed it.

Collection thumbnails will now update with their new pending tags correctly.

Some wx.Panel vs wx.Window issues and nomenclature resolved.

Full Page decouple. Now the window to the left is a seperate, much neater class. It and the media/preview windows talk to each other in a clever way that means I can make new kinds of page much faster. It scrolls, too.

Thumbnails resize will tell the thumbnail window just to redraw itself, not refresh the entire query.

�system:local� and �system:not local� are now mutually exclusive (will swap each other out, rather than overlap), like inbox and archive.

A little neatening of the system predicate code.

�system:similar_to� RMB menu entry fixed for some cases.

There is a new log window. Go view->new log page to see it. It doesn�t show much right now, but you can see synchro successes and errors pop up as they happen.

There is now a popup button on the import dialog and the URL downloader page that does �advanced import options�. You can now tell these import processes to archive everything and/or ignore files below a certain size or resolution. By default, the client will not import anything <5KB or <50x50px. You can set no limit.

Flash windows will now own all keystrokes as long as the mouse cursor is over the flash. If you are browsing/filtering, just move your mouse over or off the flash window to make sure your return/page down/spacebar/escape/whatever is interpreted as you want.

The URL downloader will not download the same file twice in a row, so you can add a thread URL several times in a row, and it�ll only download the newest files. The behaviour of this is affected by the �exclude deleted files� checkbox in the client�s options. I�ll implement thread watching a bit later.

version 26

The double unnamespaced count bug ( �system:blah (70)�, �blah (140)� ) in autocomplete should be gone.

Some rejiggering of how the statusbar updates means active import windows can�t spam the statusbar when they don�t have focus.

The count of pending tags is more accurate and uniform. When it says (+3), it means for three files, not in three cases.

Tag counts overall are better. When it says �diaper vore (17)�, you should be getting 17 files if you enter that search.

GUI tag repo precedence is slightly more accurate, compared to what the db thinks it should be.

I nailed down why system:mime wasn�t working sometimes, and now it should work all the time.

I discovered some problems with repo-side bandwidth tracking. I fixed a couple bits, but have notes to look at it more.

In v25, if system:limit was greater than the search population, it threw an error! Whoops! Fixed now.

The post-filtering thumbnail problem is totally fixed, as are a couple other instances when thumbs would sometimes not draw.

The fullscreen change-image-jittering-around problem is fixed, I think.

Fullscreen archive works again. Not sure why/when it broke.

Importing with archive checkbox ticked sometimes didn�t archive, lol.

A bunch of decoupling of gui elements, making for much neater code and less gui blocking.

Decoupling of import from hdd as well.

Query is now asynchronous! If you do a massive query on a slow computer, you won�t lose the gui! You can go check other tabs, whatever, and the query will process in the background, rendering on the page you started it when it is done.

A number of cpu-heavy numbers are now cleverly cached, so system predicate fetch happens much faster, especially on 100,000+ image dbs.

Some �collect� code is sped up and made generally less bonkers.

I wrote a couple new classes for my most common gui elements.

There is now an option in file->options to change the number of characters before autocomplete will go search the db for tags! If you have a slow computer and a huge db, you can set it to 2, 3 or even more chars. Default is 1.

Dialogs that ask you questions will now usually center on their parent window.

The current selection tagsbox won�t show ( �blah (12)�, �blah (+3)� ), it�ll combine into �blah (12) (+3) (-1)�. With the + meaning pending and the - meaning petitioned.

Autocomplete now respects which of the include_current and include_pending buttons are green! It�ll only search current/pending as you wish! And the numbers will now be accurate!

My new question dialog with the yes/no buttons hooks the Escape key to mean no.

I don�t use it much yet, but I have a new message dialog too.

version 25

Thumbnails should load just a bit quicker.

Made some dict + set code neater

A new end-of-filtering dialog makes the choices a bit more obvious

When you exit from fullscreen/filter view, the main thumbnail view it was based on should select (and scroll to!) where you were. But filtering is less reliable, since you may have just deleted the last thing you were looking at!

Better scrolling in thumbnail view

Scrolling in thumbnail view will follow the current selection, so the focussed media will always be visible.

Double right-clicking in filter now deletes instead of keeps (typo)

Slightly less flickering when processing repo updates

Made some form code neater

A lot of the thumbnail gui code has been decoupled and pubsub�d. It is all neater behind the scenes, meaning I�ll be able to extend it easier in future.

I completely overhauled how my pubsub works. (pubsub is a messaging system inside the code). It is more flexible and easier for me to work with.

system:mime was not working in v23, not sure why. It�s fixed now.

Some small speedup + neatening in db queries.

You can now scroll to the end of a thumbnail page by pressing End. WARNING: If you have like 10,000 thumbs in the current pane, it is going to have to load all those.

In the AutocompleteTagWrite object, which is the tag entry box in the manage tags dialog, there was a stupid bug that made many requests more expensive. It�s fixed.

Autocomplete now works off one character�s input.

I�ve swapped a bunch of expensive objects with easier staticbitmaps for now

some A/C ordering changes

version 24

imageboard dumper prototype

some db update stuff

version 23

A typo was disallowing petition rescinds in certain cases.

I�ve rearranged some of the RMB menu separators to reduce filter/archive misclicks.

Tag pending upload has more accurate GUI response.

You can now find similar images from RMB menu.

Petition approvals are mirrored on an admin�s client.

The gui now accepts that admins can overrule deleted content.

Couple tiny URL downloader gui bugs fixed

version 22

system:similar_to

Some backend namespace stuff

Couple of gui presentation fixes

I removed the clunky inbox and archive buttons, see how you feel about it.

I tried to do ultimatelistctrl to have coloured tags (like a *booru), but it was too buggy and slow for the client�s purposes, so I�ll probably have to write my own control.

system:mime exists, now. Not many mimes, yet, but have a play with it, see if you like it.

version 21

On opening the URL downloader, the textbox input focuses.

The URL downloader has some better update texts. It pauses nicely, etc�

The URL downloader now orders urls in the order they are parsed. I forgot I was throwing the links in a set, losing all order!

You won�t see dupes in the URL downloaders� mediaresultswindow.

Accounts should now correctly refresh on version update.

When accounts fail to sync, it dumps the error in the log.

version 20

The preview pane should be a little less schizophrenic.

Some code cleaning up behind the scenes.

Thumbnail download optimisation if they already exist locally.

Icons will show/hide in fullscreen with mouse movement, just like 21/123.

Weird archiving bug on import is fixed. It was display only.

Manage tags dialog now works in fullscreen, lol.

system:not_uploaded_to: now works

services->review services now loads muuuuuch faster (after the first time)

There was a typo in setting up initial server�s accounts. Email me if you can�t make new admins.

Importing is more streamlined, with more shared code.

Importing with tags dialog happens more naturally in the import process. You can now do it from drag and drop imports!

I reordered the default collection dropdown. Your default may be changed, so check your options.

You can now fetch an account by a deleted file (hash).

Thumbnails� tags now obey tag precedence for deleted stuff. I hope.

Modify Account Dialog now does its stuff.

Collections thumbnails now show how many files are in the collection, just next to the icon.

I�ve added to and edited the help extensively, replacing screenshots and updating the text.

version 19

drop librarium nomenclature

More intelligent prefetch in fullscreen - things should load muuuuuch faster

A little more intelligent sum() calcs in autocomplete - non-namespaced tags now sum all their namespaced versions as well

More rigorous client db transactions, just to make things more pretty and reliable.

Moved db files to /db, logs to /logs

Made inroads to getting review services to load faster, but version 20 will have more here.

I�ve updated the frame view so the file info is draw on top of the image. It�s a hack for now, inside the image-frame, but it is the only solution for wx�s annoying �no overlapping� policy.

version 18

Some misc gui image info bugs fixed

Review services dialog is less pubsub buggy

Fullscreen image prefetch does more and hopefully more intelligently

Entering a blank tag to quit tag dialog was broke when I updated autocomplete textctrls. I�ve fixed it.

Petition page is fixed, so I can finally clear out the tag petitions!

F10 was broke when the lib had deleted files lol! Fixed it.

Review services is more responsive to repo permissions updates.

Flash files are now correctly put in the inbox on import.

Synchronise Repositories daemon no longer takes ten gui-blocking seconds to process every update! This was an annoying slowdown, but I figured out what it was and reduced it to <1s with a quick patch. The problem will return with private tag repos, but I�ll deal with that when I come to it.

Synchronise Accounts daemon has some better pubsub so it fires more intelligently. This in turn affects several other daemons, which should all be more responsive; e.g. when your permissions change, your pending thumbs/files should start downloading.

Improved delete orphans logic for the fiftieth time! It is great now, and won�t ever bug you in the gui; it runs invisibly on shutdown and usually only takes a few secs to do its job.

Some of the gui updates triggered by the daemons are optimised so they block the gui less. Multiple stuff is bundled into single updates, those updates are less buggy, and the gui catches them better. Resetting the public tag repo will no longer lock up a slow system.

A few versions ago I changed the way repos were registered with the gui, and I�ve fixed some of the resultant daemon-related errors. This mostly affects admins, which mostly means me!

Download Thumbnails daemon was getting pretty slow for large dbs (100k+ files) because of an inefficient db query. This has been cleared up.

I banged my head against the wall for a couple of hours getting nested notebooks to work, and I am now a better person for it. Review and Manage Services dialogs feature better panel management as a result. Their panels are also much richer, reporting loads more data about the respective service. You can now see how many mappings the public tag repo has without having to delve into sqlite db browser.

I have unified panel and frame background colouring away from wx�s bonkers idea of what things should be coloured. It should mean that everything is properly grey, but please tell me if your gui turns all black or transparent or whatever.

21457 is now 21,457 in pretty much all cases where it is appropriate.

Fixed a bug with wx gauges not liking large ranges because of signed short or whatever underlying var the C code was using to store it. (That should have been done at the wx-python level, but whatever.)

�localhost� resolves to 127.0.0.1 when editing a service, mainly because windows freaks out at the former, deciding to take a couple seconds longer to make any raw connection.

Dialogs and frames have better, multi-monitor-friendly positioning.

Flash files are loaded from a temp folder beneath the install dir, rather than lib�s server, for various reasons. This temp dir also manages file copy/paste operations. I may move this back to the OS�s approved temp folder, but we�ll see.

version 17

Tag negation �-tag� works again

You can now cancel a filter on the first image

Some weird filenames that were breaking the new import page no longer do

Autocomplete (either when searching for tags or adding new ones in) will now show the number of files tagged with that tag after the tag e.g. here

Manage tag dialog now pops up slightly to the upper left, rather than the center (default) of the screen, so you can see what you are tagging. I may make this more intelligent in future, so it remembers where you last left it.

I changed a couple small things in help regarding the new import.

version 16

Numtags system predicate now takes into account which of current/pending tags buttons you have active.

Thumbnail downloads fire more reliably.

Downloading a file was causing funky thumbnail draw, which seems to be fixed.

Delete Orphans daemon only fires on application close now, so post-filter db lag is gone.

Daemons now use events, not conditions! lololol, I�m an idiot who can�t scroll down a library doc. For the longest time I was like �I wish �threading� had events!� and it does.

Some account checks were missing in tag upload, but are now there.

A gigantic db rewrite that altered pretty much every table and query.

Calculation of tags for current selection now support collections, and is fast.

Was deleting a repo broke in 15? If it was, sorry! Didn�t catch it, but it�s fixed with the new system now.

Pubsub can do more, and is a little less �argh, I think it works like that�, and more OOP

The way tags are held by a thumb can support multiple tag servers (but only does public tag repo for now)

Repos are now a subclass of �service�. I�ll do more on this in the next version, including some nomenclature changes in the gui.

Imports are now asynchronous! There�s a new import page that�ll open on any import. It�s a little jittery, but it all works. If you import 3000 tiny files, your gui may lock up for a few secs since it�ll be able to import faster than it updates the gui.

version 15

manage tags dialog focus bug hotfix

version 14

Selection�s tags box now shows collections� singletons� tags

Autocomplete now sources from the current thumbnails whenever it can (faster than db), and does so intelligently

Couple of small display bugs fixed

Some delete logic approved

Repos do some more efficient inserts on POST content

Petition optimisation, for auto-soft-approval before update gen

Dialog parents, dialog parents everywhere

DB cleanup of a couple of old methods

Autocomplete Write (when adding tags in manage tags dialog) now sources from the entire network�s tags (not just for the files on your hard drive) when you type enough letters to stump it. i.e. If you have no files tagged �evangelion�, but you do have a file tagged �eve�, it�ll show �eve� if you type �ev�, but if you type �eva�, it�ll requery the db more widely, and give you �evangelion�.

system:ratio for searching for wallpapers or whatever. Combine with system:width>x for extra utility!

You can now hit F3 (manage tags) in fullscreen, whether browsing or filtering.

version 13

a couple of cpu-intensive DAEMONS schedule themselves better. (less slowdown after a big filter)

image rendering prefetch improved

smoother image rendering to screen (especially when zooming), less flickering

petition approval has smoother, less buggy workflow

new hashes and tags are added to the db muuuuuuuuch more efficiently. most public tag repo updates happen in a second or two, rather than 30+

some of the internal commands are streamlined together into a single class

ctrl-s fixed

clear pending file repos pubsub fixed

ctrl-r now shows/hides splitters

thumbnail delete logic is better

select logic is better

deleted_remote_files bug fixed

headache-inducing weakref repositories __eq__ bug in pubsub fixed

�with� keyword all over the place now. turns out wx has a bunch of objects with support it (including dialogs!)

gridsizers are now all flexgridsizers

the list of tags on the current selection now has a weight attached like �flower (23)� to show how many of the current selection have that tag. it doesn�t support collections� weight yet.

revised some ambiguous repo error messages

you can now have a main window with no pages open

right click menus on a selection are vastly improved (lots more info), but still a way to go.

on the import files dialog, you can now specify to auto-archive rather than send to inbox

thumbnails now know where they are uploaded/pending/petitioned to and will show that info with different icons

escape key now deselects

manage repos dialog is completely revised. comments appreciated.

version 12

You can now tell lib to search �current tags� only, �pending tags� only, or both.

Tag search is more efficient.

Search in general is more efficient.

Search code is more beautiful and decoupled.

Tags are stored in memory more beautifully, and a similar class holds repositories for thumbs also.

Age predicate is fixed (twice!)

Server code is more beautiful. DB-Server interface is decoupled into a mixin class to make it easier to understand and manage.

Icons for pending and petitioned files, and for �currently downloading�.

Prefetch fixed, so nearby fullscreen images load just in case you want to see them.

Fixed a redundant pending tags bug

Fixed a swf tag petitions bug

On boot, temp folder is cleared.

The �current tags� window is now union on the current selection, not intersect. If nothing is selected, it shows all the tags of the current selection. In a later version, I�ll order this list by count-weight.

All (I think) tag updates are synchronous-ish with the gui. As updates come in from the public repo, thumbs will update themselves appropriately. Same goes for repositories.

Gui-code tidy-up

On deleting a local file, preview clears.

version 11

swf support, lol

You can now backup repositories from the admin menu. Info in help.

Under the help menu, a quick way to set up the beta repositories� credentials.

Account sync was full of bugs, and is now less so.

When you go fullscreen, the preview window should go blank, which should save a little cpu time when you are looking at gifs and your ears when you are looking at swfs.

Auto-complete is muuuuuch faster than it was. Also more correct. Still more work to do, though.

Librarium hosts an http server on 45874, which supports GET /file and /thumbnail just like a file repo. Utterly prototype, but you can test it right in your browser if you like. Go http://127.0.0.1:45874/file?hash=whatever when lib is running.

Ctrl-T opens a new local page/tab, Ctrl-W closes the currently active

Image display in preview and fullscreen has been revamped, and is now both a little buggy and much better.

Bandwidth tracking both server- and client-side should be more reliable.

Delete lag (the slowdown after you boot/delete something) should be reduced.

Query lag is significantly reduced, and new operators for predicates have been added. You can now go � on a bunch of values like size and width and duration to get �approximately equal to�, which actually means plus or minus about five percent.

A stupid error on file upload is fixed. If you�ve been having problems seeing the files you uploaded in preview, try resetting the respective file repository.

A few daemon bug fixes.

version 10

hotfix for tag updates

version 9

Pressing a cursor key when no thumb is focussed no longer throws an exception.

Hitting enter on a blank input in the manage tags dialog now quickly oks the dialog.

Annoying splitter exception on import is gone. It was a one-line fix, because I was being stupid.

All searches should be a little faster. Mostly effects searches with 1000+ results.

Modal dialogs (in particular) should be just a little more responsive now. Less stuff is being dumped on the wx gui thread, and all db handling is more parallel.

Slow petition approval fixed, made almost instant.

Image rendering has been revamped. It should be less stuttery now, but there is still more work to do.

Some pending tag update logic was fixed.

If the focused thumb goes from �not local� to �local�, its preview will load. (its on my to-do to do the opposite: clearing preview on delete in remote search.

DB daemons rewritten completely, so now a whole bunch of stuff flushes more efficiently, more reliably, and with less gui-blocking. Things happen smoother now.

Can now create repository database backups via librarium. Sending the command (via the admin menu) tells the repo to lock and copy its .db files to .db.backup. I�ll update help to explain this more.

Can now copy files from within librarium to the clipboard! Just select from the right-click menu or press Ctrl-C, and your selection will be copied to clipboard. You can then paste to any directory, for quick export to anywhere. The export command remains for exporting thousands of files.

version 8

Fixed a missing transaction on repo db update.

Export path can now be on other hard drives. You�ll probably want to keep it on the same drive if you have lib on a usb stick, though.

Fixed a really stupid �<� for �>� typo that was causing a ton of gui blocking and repository slowdown.

File and thumbnail downloads are much snappier.

One person had a very odd error while trying to open manage repos. I don�t understand what actually caused the traceback to happen, but I think I fixed it nonetheless.

inbox/archive buttons above the searchbox are now mutually exclusive.

Fixed the version number in lib�s about window.

Fixed the preview window so it remembers its last size properly. Also reduced its min size so you can make it tiny.

Fixed the problem people were having with admin rights. See this post if you were having problems.

The RegisterRepoRequest that was filling people�s logs on upload should be fixed now. Turns out I didn�t understand how wx.CallAfter works during modal dialogs! I threw in a horrible fix, but it works. I�ll revisit it.

version 7

db changes are done

fixed a bug with posting tag petitions

when typing tags in, autocomplete now harvests from pending tags as well as current

the regex dialog�s tag entry is now autocomplete

leading/following whitespace removal for tags, and multi-whitespace -> single

a little server-side pubsub neatening up

when importing via dialog, there is now a checkbox to ignore subdirectories

Added a quick readme.txt in the base_dir for people who choose to extract rather than install. (Everyone should check out the help at least once)

version 3->6 (never released)

after much feedback, completely rewrite db

general improvements to gui

version 2

general fixes and improvements

version 1

first true metadata beta

shit actually works

versions 0.91 -> 0.99

move to python and wx, and an application- rather than browser-based model

building on concepts

scrapping p2p, turning to client-server

early versions of beta

made db, playing around with LAMP

an initial p2p framework started