code:
		<title>privacy</title>
		<link href="hydrus.ico" rel="shortcut icon" />
		<link href="style.css" rel="stylesheet" type="text/css" />

privacy

Repositories never know what you are searching for. The client synchronises (copies) the repository's entire file or mapping list to its internal database, and does its own searches over those internal caches, all on your hard drive. It never sends search queries outside your own computer, nor does it log what you do look for. Your searches are your business, and no-one else's.

Repositories know nothing more about your client than they can infer, and the software usually commands them to forget as much as possible as soon as possible. Specifically:

					tag repository
					file repository

					upload mappings
					download mappings
					upload file
					download file

					Account is linked to action
					Yes
					No
					Yes
					No

					IP address is remembered
					No
					No
					Maybe
					No

i.e:

If you download anything from any repository, your accessing it will not be recorded. A running total of your approximate bandwidth and number of queries made for the current month is kept so the respective administrator can combat leechers.

If you upload a mapping to a tag repository, your anonymous account is linked so the administrator can quickly revoke all of a rule-breaker's contributions. Your IP address is forgotten.

If you upload a file to a file repository, your anonymous account is linked so the administrator can quickly revoke all of a rule-breaker's contributions. Your IP may be recorded, depending on whether the repository's administrator has decided to enable ip upload-logging or not.

Furthermore:

Administrators for a particular repository can see which accounts uploaded what. If IP addresses are available, they can discover which IP uploaded a particular file, and when.

Repositories do not talk to each other.

All accounts are anonymous. Repositories do not know any of their accounts' access keys and cannot produce them on demand; they can determine whether a particular access key refers to a particular account, but the access keys themselves are all irreversibly hashed inside the repository database.

There are of course some clever exceptions. If you tag a file three years before it surfaces on the internet, someone with enough knowledge will be able to infer it was most likely you who created it. If you set up a file repository for just a friend and yourself, it becomes trivial by elimination to guess who uploaded the NarutoXSonichu shota diaper fanon. If you sign up for a file repository that hosts only certain stuff and rack up a huge bandwidth record for the current month, anyone who knows that and also knows the account is yours alone will know basically what you were up to.

Note also that the file repository code is freely available and entirely mutable. If someone wants to put the time in, they can create a file repository that looks from the outside like any other but nonetheless logs the IP and nature of every request. Just make sure you trust the person running the repository. (And make sure they suck at programming python!)

Even anonymised records can reveal personally identifying information. Don't trust anyone who plans to release maps of accounts -> files or accounts -> mappings, even for some benevolent academic purpose.


code:
2016/09/18 13:47:09: Profiling write content_updates
2016/09/18 13:47:09: Stats

         278 function calls in 0.164 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       25    0.163    0.007    0.163    0.007 {method 'execute' of 'sqlite3.Cursor' objects}
        9    0.000    0.000    0.000    0.000 {method 'executemany' of 'sqlite3.Cursor' objects}
        1    0.000    0.000    0.001    0.001 ClientDB.py:8072(_UpdateMappings)
        1    0.000    0.000    0.002    0.002 ClientDB.py:5519(_ProcessContentUpdates)
        1    0.000    0.000    0.164    0.164 HydrusDB.py:451(_ProcessJob)
        4    0.000    0.000    0.000    0.000 ClientDB.py:1738(_CacheSpecificMappingsPendMappings)
        7    0.000    0.000    0.000    0.000 {method 'join' of 'str' objects}
        3    0.000    0.000    0.000    0.000 _core.py:16750(CallAfter)
        4    0.000    0.000    0.000    0.000 ClientDB.py:1695(_CacheSpecificMappingsFilterHashIds)
        3    0.000    0.000    0.000    0.000 {wx._core_.PostEvent}
        8    0.000    0.000    0.000    0.000 ClientDB.py:1014(GenerateSpecificMappingsCacheTableNames)
        2    0.000    0.000    0.000    0.000 ClientDB.py:4621(_GetServiceIds)
        1    0.000    0.000    0.000    0.000 HydrusTags.py:150(CleanTag)
        7    0.000    0.000    0.000    0.000 HydrusData.py:945(SplayListForDB)
        3    0.000    0.000    0.000    0.000 HydrusPubSub.py:137(pub)
        3    0.000    0.000    0.000    0.000 {method 'sub' of '_sre.SRE_Pattern' objects}
        2    0.000    0.000    0.000    0.000 collections.py:458(__init__)
        1    0.000    0.000    0.000    0.000 threading.py:373(notify)
        1    0.000    0.000    0.000    0.000 ClientDB.py:2970(_GetHashIds)
        3    0.000    0.000    0.000    0.000 _core.py:7583(__init__)
        1    0.000    0.000    0.000    0.000 ClientDB.py:1498(_CacheCombinedFilesMappingsUpdate)
       16    0.000    0.000    0.000    0.000 HydrusData.py:945(<genexpr>)
        4    0.000    0.000    0.000    0.000 re.py:230(_compile)
        1    0.000    0.000    0.000    0.000 ClientDB.py:4058(_GetNamespaceIdTagId)
        1    0.000    0.000    0.000    0.000 ClientDB.py:4607(_GetServiceId)
        4    0.000    0.000    0.000    0.000 {method 'fetchone' of 'sqlite3.Cursor' objects}
        3    0.000    0.000    0.000    0.000 re.py:148(sub)
        3    0.000    0.000    0.000    0.000 HydrusController.py:142(pub)
        3    0.000    0.000    0.000    0.000 {wx._core_.new_PyEvent}
        3    0.000    0.000    0.000    0.000 ClientController.py:699(NotifyPubSubs)
        1    0.000    0.000    0.000    0.000 threading.py:285(__enter__)
        3    0.000    0.000    0.000    0.000 {wx._core_.GetApp}
        1    0.000    0.000    0.000    0.000 threading.py:576(set)
        1    0.000    0.000    0.002    0.002 ClientDB.py:8645(_Write)
        1    0.000    0.000    0.000    0.000 ClientDB.py:4510(_GetService)
        3    0.000    0.000    0.000    0.000 _core.py:8421(GetApp)
       25    0.000    0.000    0.000    0.000 {len}
        2    0.000    0.000    0.000    0.000 HydrusDB.py:317(_GetRowCount)
        1    0.000    0.000    0.000    0.000 HydrusData.py:1885(PutResult)
       10    0.000    0.000    0.000    0.000 {method 'append' of 'list' objects}
        1    0.000    0.000    0.000    0.000 HydrusData.py:1028(ToUnicode)
        1    0.000    0.000    0.000    0.000 threading.py:300(_is_owned)
        1    0.000    0.000    0.000    0.000 threading.py:400(notifyAll)
        3    0.000    0.000    0.000    0.000 {wx._core_.PyEvent__SetSelf}
        2    0.000    0.000    0.000    0.000 {method 'replace' of 'unicode' objects}
        3    0.000    0.000    0.000    0.000 {wx._core_.Event_SetEventType}
        2    0.000    0.000    0.000    0.000 collections.py:528(update)
        3    0.000    0.000    0.000    0.000 HydrusDB.py:528(pub_after_commit)
        5    0.000    0.000    0.000    0.000 {method 'difference' of 'set' objects}
        1    0.000    0.000    0.000    0.000 ClientDB.py:1000(GenerateMappingsTableNames)
        1    0.000    0.000    0.000    0.000 ClientDB.py:8265(<setcomp>)
        1    0.000    0.000    0.000    0.000 ClientDB.py:8124(<setcomp>)
        1    0.000    0.000    0.000    0.000 {method 'match' of '_sre.SRE_Pattern' objects}
        1    0.000    0.000    0.000    0.000 re.py:138(match)
        1    0.000    0.000    0.164    0.164 <string>:1(<module>)
        3    0.000    0.000    0.000    0.000 _core.py:7590(_SetSelf)
        3    0.000    0.000    0.000    0.000 {wx._core_.PyEvent_swiginit}
        2    0.000    0.000    0.000    0.000 ClientDB.py:1748(<genexpr>)
        1    0.000    0.000    0.000    0.000 ClientDB.py:8686(pub_content_updates_after_commit)
        1    0.000    0.000    0.000    0.000 {method 'update' of 'set' objects}
        1    0.000    0.000    0.000    0.000 threading.py:288(__exit__)
        3    0.000    0.000    0.000    0.000 {method 'items' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 HydrusTags.py:135(CheckTagNotEmpty)
        1    0.000    0.000    0.000    0.000 ClientDB.py:996(GenerateCombinedFilesMappingsCacheTableName)
        1    0.000    0.000    0.000    0.000 {method 'lower' of 'unicode' objects}
        1    0.000    0.000    0.000    0.000 HydrusData.py:1789(ToTuple)
        3    0.000    0.000    0.000    0.000 _core.py:4972(SetEventType)
        2    0.000    0.000    0.000    0.000 {method 'intersection' of 'set' objects}
        1    0.000    0.000    0.000    0.000 {method 'add' of 'set' objects}
        3    0.000    0.000    0.000    0.000 _core.py:8403(PostEvent)
        2    0.000    0.000    0.000    0.000 {isinstance}
        2    0.000    0.000    0.000    0.000 {method 'union' of 'set' objects}
        1    0.000    0.000    0.000    0.000 ClientDB.py:8123(<setcomp>)
        1    0.000    0.000    0.000    0.000 ClientDB.py:8105(<setcomp>)
        1    0.000    0.000    0.000    0.000 ClientDB.py:8264(<setcomp>)
        2    0.000    0.000    0.000    0.000 ClientDB.py:8121(<genexpr>)
        3    0.000    0.000    0.000    0.000 {hasattr}
        1    0.000    0.000    0.000    0.000 HydrusData.py:1883(IsSynchronous)
        2    0.000    0.000    0.000    0.000 ClientDB.py:8206(<genexpr>)
        1    0.000    0.000    0.000    0.000 ClientDB.py:8102(<setcomp>)
        2    0.000    0.000    0.000    0.000 ClientDB.py:8256(<genexpr>)
        2    0.000    0.000    0.000    0.000 ClientDB.py:8120(<genexpr>)
        2    0.000    0.000    0.000    0.000 {built-in method from_iterable}
        2    0.000    0.000    0.000    0.000 ClientDB.py:1504(<genexpr>)
        1    0.000    0.000    0.000    0.000 HydrusData.py:1858(GetCallableTuple)
        2    0.000    0.000    0.000    0.000 ClientData.py:1338(GetServiceType)
        1    0.000    0.000    0.000    0.000 threading.py:64(_note)
        1    0.000    0.000    0.000    0.000 ClientDB.py:8255(<genexpr>)
        2    0.000    0.000    0.000    0.000 ClientDB.py:1506(<genexpr>)
        1    0.000    0.000    0.000    0.000 HydrusData.py:1881(GetType)
        1    0.000    0.000    0.000    0.000 ClientDB.py:8112(<setcomp>)
        1    0.000    0.000    0.000    0.000 {method 'remove' of 'list' objects}
        3    0.000    0.000    0.000    0.000 {callable}
        1    0.000    0.000    0.000    0.000 {method '__enter__' of 'thread.lock' objects}
        2    0.000    0.000    0.000    0.000 ClientDB.py:1502(<genexpr>)
        1    0.000    0.000    0.000    0.000 ClientDB.py:8109(<setcomp>)
        1    0.000    0.000    0.000    0.000 {method 'acquire' of 'thread.lock' objects}
        1    0.000    0.000    0.000    0.000 ClientDB.py:8208(<setcomp>)
        2    0.000    0.000    0.000    0.000 collections.py:479(__missing__)
        1    0.000    0.000    0.000    0.000 {method 'release' of 'thread.lock' objects}
        1    0.000    0.000    0.000    0.000 {method '__exit__' of 'thread.lock' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

2016/09/18 13:47:09: Callers

   Ordered by: internal time

Function                                                      was called by...
                                                                  ncalls  tottime  cumtime
{method 'execute' of 'sqlite3.Cursor' objects}                <-       4    0.000    0.000  ClientDB.py:1695(_CacheSpecificMappingsFilterHashIds)
                                                                       2    0.000    0.000  ClientDB.py:1738(_CacheSpecificMappingsPendMappings)
                                                                       1    0.000    0.000  ClientDB.py:2970(_GetHashIds)
                                                                       2    0.000    0.000  ClientDB.py:4058(_GetNamespaceIdTagId)
                                                                       1    0.000    0.000  ClientDB.py:4607(_GetServiceId)
                                                                       2    0.000    0.000  ClientDB.py:4621(_GetServiceIds)
                                                                      11    0.001    0.001  ClientDB.py:8072(_UpdateMappings)
                                                                       2    0.162    0.162  HydrusDB.py:451(_ProcessJob)
{method 'executemany' of 'sqlite3.Cursor' objects}            <-       3    0.000    0.000  ClientDB.py:1498(_CacheCombinedFilesMappingsUpdate)
                                                                       1    0.000    0.000  ClientDB.py:1738(_CacheSpecificMappingsPendMappings)
                                                                       5    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
ClientDB.py:8072(_UpdateMappings)                             <-       1    0.000    0.001  ClientDB.py:5519(_ProcessContentUpdates)
ClientDB.py:5519(_ProcessContentUpdates)                      <-       1    0.000    0.002  ClientDB.py:8645(_Write)
HydrusDB.py:451(_ProcessJob)                                  <-       1    0.000    0.164  <string>:1(<module>)
ClientDB.py:1738(_CacheSpecificMappingsPendMappings)          <-       4    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
{method 'join' of 'str' objects}                              <-       7    0.000    0.000  HydrusData.py:945(SplayListForDB)
_core.py:16750(CallAfter)                                     <-       3    0.000    0.000  ClientController.py:699(NotifyPubSubs)
ClientDB.py:1695(_CacheSpecificMappingsFilterHashIds)         <-       4    0.000    0.000  ClientDB.py:1738(_CacheSpecificMappingsPendMappings)
{wx._core_.PostEvent}                                         <-       3    0.000    0.000  _core.py:8403(PostEvent)
ClientDB.py:1014(GenerateSpecificMappingsCacheTableNames)     <-       4    0.000    0.000  ClientDB.py:1695(_CacheSpecificMappingsFilterHashIds)
                                                                       4    0.000    0.000  ClientDB.py:1738(_CacheSpecificMappingsPendMappings)
ClientDB.py:4621(_GetServiceIds)                              <-       2    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
HydrusTags.py:150(CleanTag)                                   <-       1    0.000    0.000  ClientDB.py:4058(_GetNamespaceIdTagId)
HydrusData.py:945(SplayListForDB)                             <-       4    0.000    0.000  ClientDB.py:1695(_CacheSpecificMappingsFilterHashIds)
                                                                       2    0.000    0.000  ClientDB.py:4621(_GetServiceIds)
                                                                       1    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
HydrusPubSub.py:137(pub)                                      <-       3    0.000    0.000  HydrusController.py:142(pub)
{method 'sub' of '_sre.SRE_Pattern' objects}                  <-       3    0.000    0.000  re.py:148(sub)
collections.py:458(__init__)                                  <-       2    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
threading.py:373(notify)                                      <-       1    0.000    0.000  threading.py:400(notifyAll)
ClientDB.py:2970(_GetHashIds)                                 <-       1    0.000    0.000  ClientDB.py:5519(_ProcessContentUpdates)
_core.py:7583(__init__)                                       <-       3    0.000    0.000  _core.py:16750(CallAfter)
ClientDB.py:1498(_CacheCombinedFilesMappingsUpdate)           <-       1    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
HydrusData.py:945(<genexpr>)                                  <-      16    0.000    0.000  {method 'join' of 'str' objects}
re.py:230(_compile)                                           <-       1    0.000    0.000  re.py:138(match)
                                                                       3    0.000    0.000  re.py:148(sub)
ClientDB.py:4058(_GetNamespaceIdTagId)                        <-       1    0.000    0.000  ClientDB.py:5519(_ProcessContentUpdates)
ClientDB.py:4607(_GetServiceId)                               <-       1    0.000    0.000  ClientDB.py:5519(_ProcessContentUpdates)
{method 'fetchone' of 'sqlite3.Cursor' objects}               <-       1    0.000    0.000  ClientDB.py:2970(_GetHashIds)
                                                                       2    0.000    0.000  ClientDB.py:4058(_GetNamespaceIdTagId)
                                                                       1    0.000    0.000  ClientDB.py:4607(_GetServiceId)
re.py:148(sub)                                                <-       3    0.000    0.000  HydrusTags.py:150(CleanTag)
HydrusController.py:142(pub)                                  <-       3    0.000    0.000  HydrusDB.py:451(_ProcessJob)
{wx._core_.new_PyEvent}                                       <-       3    0.000    0.000  _core.py:7583(__init__)
ClientController.py:699(NotifyPubSubs)                        <-       3    0.000    0.000  HydrusPubSub.py:137(pub)
threading.py:285(__enter__)                                   <-       1    0.000    0.000  threading.py:576(set)
{wx._core_.GetApp}                                            <-       3    0.000    0.000  _core.py:8421(GetApp)
threading.py:576(set)                                         <-       1    0.000    0.000  HydrusData.py:1885(PutResult)
ClientDB.py:8645(_Write)                                      <-       1    0.000    0.002  HydrusDB.py:451(_ProcessJob)
ClientDB.py:4510(_GetService)                                 <-       1    0.000    0.000  ClientDB.py:5519(_ProcessContentUpdates)
_core.py:8421(GetApp)                                         <-       3    0.000    0.000  _core.py:16750(CallAfter)
{len}                                                         <-       4    0.000    0.000  ClientDB.py:1738(_CacheSpecificMappingsPendMappings)
                                                                       1    0.000    0.000  ClientDB.py:2970(_GetHashIds)
                                                                       6    0.000    0.000  ClientDB.py:5519(_ProcessContentUpdates)
                                                                       9    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
                                                                       2    0.000    0.000  collections.py:458(__init__)
                                                                       2    0.000    0.000  collections.py:528(update)
                                                                       1    0.000    0.000  threading.py:400(notifyAll)
HydrusDB.py:317(_GetRowCount)                                 <-       1    0.000    0.000  ClientDB.py:1738(_CacheSpecificMappingsPendMappings)
                                                                       1    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
HydrusData.py:1885(PutResult)                                 <-       1    0.000    0.000  HydrusDB.py:451(_ProcessJob)
{method 'append' of 'list' objects}                           <-       1    0.000    0.000  ClientDB.py:5519(_ProcessContentUpdates)
                                                                       3    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
                                                                       3    0.000    0.000  HydrusDB.py:528(pub_after_commit)
                                                                       3    0.000    0.000  HydrusPubSub.py:137(pub)
HydrusData.py:1028(ToUnicode)                                 <-       1    0.000    0.000  HydrusTags.py:150(CleanTag)
threading.py:300(_is_owned)                                   <-       1    0.000    0.000  threading.py:373(notify)
threading.py:400(notifyAll)                                   <-       1    0.000    0.000  threading.py:576(set)
{wx._core_.PyEvent__SetSelf}                                  <-       3    0.000    0.000  _core.py:7590(_SetSelf)
{method 'replace' of 'unicode' objects}                       <-       2    0.000    0.000  HydrusTags.py:150(CleanTag)
{wx._core_.Event_SetEventType}                                <-       3    0.000    0.000  _core.py:4972(SetEventType)
collections.py:528(update)                                    <-       2    0.000    0.000  collections.py:458(__init__)
HydrusDB.py:528(pub_after_commit)                             <-       1    0.000    0.000  ClientDB.py:5519(_ProcessContentUpdates)
                                                                       2    0.000    0.000  ClientDB.py:8686(pub_content_updates_after_commit)
{method 'difference' of 'set' objects}                        <-       5    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
ClientDB.py:1000(GenerateMappingsTableNames)                  <-       1    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
ClientDB.py:8265(<setcomp>)                                   <-       1    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
ClientDB.py:8124(<setcomp>)                                   <-       1    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
{method 'match' of '_sre.SRE_Pattern' objects}                <-       1    0.000    0.000  re.py:138(match)
re.py:138(match)                                              <-       1    0.000    0.000  HydrusTags.py:150(CleanTag)
<string>:1(<module>)                                          <-
_core.py:7590(_SetSelf)                                       <-       3    0.000    0.000  _core.py:7583(__init__)
{wx._core_.PyEvent_swiginit}                                  <-       3    0.000    0.000  _core.py:7583(__init__)
ClientDB.py:1748(<genexpr>)                                   <-       2    0.000    0.000  {method 'executemany' of 'sqlite3.Cursor' objects}
ClientDB.py:8686(pub_content_updates_after_commit)            <-       1    0.000    0.000  ClientDB.py:5519(_ProcessContentUpdates)
{method 'update' of 'set' objects}                            <-       1    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
threading.py:288(__exit__)                                    <-       1    0.000    0.000  threading.py:576(set)
{method 'items' of 'dict' objects}                            <-       1    0.000    0.000  ClientDB.py:5519(_ProcessContentUpdates)
                                                                       2    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
HydrusTags.py:135(CheckTagNotEmpty)                           <-       1    0.000    0.000  ClientDB.py:4058(_GetNamespaceIdTagId)
ClientDB.py:996(GenerateCombinedFilesMappingsCacheTableName)  <-       1    0.000    0.000  ClientDB.py:1498(_CacheCombinedFilesMappingsUpdate)
{method 'lower' of 'unicode' objects}                         <-       1    0.000    0.000  HydrusTags.py:150(CleanTag)
HydrusData.py:1789(ToTuple)                                   <-       1    0.000    0.000  ClientDB.py:5519(_ProcessContentUpdates)
_core.py:4972(SetEventType)                                   <-       3    0.000    0.000  _core.py:16750(CallAfter)
{method 'intersection' of 'set' objects}                      <-       2    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
{method 'add' of 'set' objects}                               <-       1    0.000    0.000  ClientDB.py:2970(_GetHashIds)
_core.py:8403(PostEvent)                                      <-       3    0.000    0.000  _core.py:16750(CallAfter)
{isinstance}                                                  <-       2    0.000    0.000  HydrusData.py:1028(ToUnicode)
{method 'union' of 'set' objects}                             <-       2    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
ClientDB.py:8123(<setcomp>)                                   <-       1    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
ClientDB.py:8105(<setcomp>)                                   <-       1    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
ClientDB.py:8264(<setcomp>)                                   <-       1    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
ClientDB.py:8121(<genexpr>)                                   <-       2    0.000    0.000  {method 'executemany' of 'sqlite3.Cursor' objects}
{hasattr}                                                     <-       3    0.000    0.000  _core.py:16750(CallAfter)
HydrusData.py:1883(IsSynchronous)                             <-       1    0.000    0.000  HydrusDB.py:451(_ProcessJob)
ClientDB.py:8206(<genexpr>)                                   <-       2    0.000    0.000  {method 'executemany' of 'sqlite3.Cursor' objects}
ClientDB.py:8102(<setcomp>)                                   <-       1    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
ClientDB.py:8256(<genexpr>)                                   <-       2    0.000    0.000  {method 'update' of 'set' objects}
ClientDB.py:8120(<genexpr>)                                   <-       2    0.000    0.000  {method 'executemany' of 'sqlite3.Cursor' objects}
{built-in method from_iterable}                               <-       2    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
ClientDB.py:1504(<genexpr>)                                   <-       2    0.000    0.000  {method 'executemany' of 'sqlite3.Cursor' objects}
HydrusData.py:1858(GetCallableTuple)                          <-       1    0.000    0.000  HydrusDB.py:451(_ProcessJob)
ClientData.py:1338(GetServiceType)                            <-       1    0.000    0.000  ClientDB.py:4510(_GetService)
                                                                       1    0.000    0.000  ClientDB.py:5519(_ProcessContentUpdates)
threading.py:64(_note)                                        <-       1    0.000    0.000  threading.py:373(notify)
ClientDB.py:8255(<genexpr>)                                   <-       1    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
ClientDB.py:1506(<genexpr>)                                   <-       2    0.000    0.000  {method 'executemany' of 'sqlite3.Cursor' objects}
HydrusData.py:1881(GetType)                                   <-       1    0.000    0.000  HydrusDB.py:451(_ProcessJob)
ClientDB.py:8112(<setcomp>)                                   <-       1    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
{method 'remove' of 'list' objects}                           <-       1    0.000    0.000  threading.py:373(notify)
{callable}                                                    <-       3    0.000    0.000  _core.py:16750(CallAfter)
{method '__enter__' of 'thread.lock' objects}                 <-       1    0.000    0.000  threading.py:285(__enter__)
ClientDB.py:1502(<genexpr>)                                   <-       2    0.000    0.000  {method 'executemany' of 'sqlite3.Cursor' objects}
ClientDB.py:8109(<setcomp>)                                   <-       1    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
{method 'acquire' of 'thread.lock' objects}                   <-       1    0.000    0.000  threading.py:300(_is_owned)
ClientDB.py:8208(<setcomp>)                                   <-       1    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
collections.py:479(__missing__)                               <-       2    0.000    0.000  ClientDB.py:8072(_UpdateMappings)
{method 'release' of 'thread.lock' objects}                   <-       1    0.000    0.000  threading.py:373(notify)
{method '__exit__' of 'thread.lock' objects}                  <-       1    0.000    0.000  threading.py:288(__exit__)
{method 'disable' of '_lsprof.Profiler' objects}              <-

2016/09/18 13:47:09: Callees

   Ordered by: internal time

Function                                                      called...
                                                                  ncalls  tottime  cumtime
{method 'execute' of 'sqlite3.Cursor' objects}                ->
{method 'executemany' of 'sqlite3.Cursor' objects}            ->       2    0.000    0.000  ClientDB.py:1502(<genexpr>)
                                                                       2    0.000    0.000  ClientDB.py:1504(<genexpr>)
                                                                       2    0.000    0.000  ClientDB.py:1506(<genexpr>)
                                                                       2    0.000    0.000  ClientDB.py:1748(<genexpr>)
                                                                       2    0.000    0.000  ClientDB.py:8120(<genexpr>)
                                                                       2    0.000    0.000  ClientDB.py:8121(<genexpr>)
                                                                       2    0.000    0.000  ClientDB.py:8206(<genexpr>)
ClientDB.py:8072(_UpdateMappings)                             ->       1    0.000    0.000  ClientDB.py:1000(GenerateMappingsTableNames)
                                                                       1    0.000    0.000  ClientDB.py:1498(_CacheCombinedFilesMappingsUpdate)
                                                                       4    0.000    0.000  ClientDB.py:1738(_CacheSpecificMappingsPendMappings)
                                                                       2    0.000    0.000  ClientDB.py:4621(_GetServiceIds)
                                                                       1    0.000    0.000  ClientDB.py:8102(<setcomp>)
                                                                       1    0.000    0.000  ClientDB.py:8105(<setcomp>)
                                                                       1    0.000    0.000  ClientDB.py:8109(<setcomp>)
                                                                       1    0.000    0.000  ClientDB.py:8112(<setcomp>)
                                                                       1    0.000    0.000  ClientDB.py:8123(<setcomp>)
                                                                       1    0.000    0.000  ClientDB.py:8124(<setcomp>)
                                                                       1    0.000    0.000  ClientDB.py:8208(<setcomp>)
                                                                       1    0.000    0.000  ClientDB.py:8255(<genexpr>)
                                                                       1    0.000    0.000  ClientDB.py:8264(<setcomp>)
                                                                       1    0.000    0.000  ClientDB.py:8265(<setcomp>)
                                                                       1    0.000    0.000  HydrusDB.py:317(_GetRowCount)
                                                                       1    0.000    0.000  HydrusData.py:945(SplayListForDB)
                                                                       2    0.000    0.000  collections.py:458(__init__)
                                                                       2    0.000    0.000  collections.py:479(__missing__)
                                                                       2    0.000    0.000  {built-in method from_iterable}
                                                                       9    0.000    0.000  {len}
                                                                       3    0.000    0.000  {method 'append' of 'list' objects}
                                                                       5    0.000    0.000  {method 'difference' of 'set' objects}
                                                                      11    0.001    0.001  {method 'execute' of 'sqlite3.Cursor' objects}
                                                                       5    0.000    0.000  {method 'executemany' of 'sqlite3.Cursor' objects}
                                                                       2    0.000    0.000  {method 'intersection' of 'set' objects}
                                                                       2    0.000    0.000  {method 'items' of 'dict' objects}
                                                                       2    0.000    0.000  {method 'union' of 'set' objects}
                                                                       1    0.000    0.000  {method 'update' of 'set' objects}
ClientDB.py:5519(_ProcessContentUpdates)                      ->       1    0.000    0.000  ClientDB.py:2970(_GetHashIds)
                                                                       1    0.000    0.000  ClientDB.py:4058(_GetNamespaceIdTagId)
                                                                       1    0.000    0.000  ClientDB.py:4510(_GetService)
                                                                       1    0.000    0.000  ClientDB.py:4607(_GetServiceId)
                                                                       1    0.000    0.001  ClientDB.py:8072(_UpdateMappings)
                                                                       1    0.000    0.000  ClientDB.py:8686(pub_content_updates_after_commit)
                                                                       1    0.000    0.000  ClientData.py:1338(GetServiceType)
                                                                       1    0.000    0.000  HydrusDB.py:528(pub_after_commit)
                                                                       1    0.000    0.000  HydrusData.py:1789(ToTuple)
                                                                       6    0.000    0.000  {len}
                                                                       1    0.000    0.000  {method 'append' of 'list' objects}
                                                                       1    0.000    0.000  {method 'items' of 'dict' objects}
HydrusDB.py:451(_ProcessJob)                                  ->       1    0.000    0.002  ClientDB.py:8645(_Write)
                                                                       3    0.000    0.000  HydrusController.py:142(pub)
                                                                       1    0.000    0.000  HydrusData.py:1858(GetCallableTuple)
                                                                       1    0.000    0.000  HydrusData.py:1881(GetType)
                                                                       1    0.000    0.000  HydrusData.py:1883(IsSynchronous)
                                                                       1    0.000    0.000  HydrusData.py:1885(PutResult)
                                                                       2    0.162    0.162  {method 'execute' of 'sqlite3.Cursor' objects}
ClientDB.py:1738(_CacheSpecificMappingsPendMappings)          ->       4    0.000    0.000  ClientDB.py:1014(GenerateSpecificMappingsCacheTableNames)
                                                                       4    0.000    0.000  ClientDB.py:1695(_CacheSpecificMappingsFilterHashIds)
                                                                       1    0.000    0.000  HydrusDB.py:317(_GetRowCount)
                                                                       4    0.000    0.000  {len}
                                                                       2    0.000    0.000  {method 'execute' of 'sqlite3.Cursor' objects}
                                                                       1    0.000    0.000  {method 'executemany' of 'sqlite3.Cursor' objects}
{method 'join' of 'str' objects}                              ->      16    0.000    0.000  HydrusData.py:945(<genexpr>)
_core.py:16750(CallAfter)                                     ->       3    0.000    0.000  _core.py:4972(SetEventType)
                                                                       3    0.000    0.000  _core.py:7583(__init__)
                                                                       3    0.000    0.000  _core.py:8403(PostEvent)
                                                                       3    0.000    0.000  _core.py:8421(GetApp)
                                                                       3    0.000    0.000  {callable}
                                                                       3    0.000    0.000  {hasattr}
ClientDB.py:1695(_CacheSpecificMappingsFilterHashIds)         ->       4    0.000    0.000  ClientDB.py:1014(GenerateSpecificMappingsCacheTableNames)
                                                                       4    0.000    0.000  HydrusData.py:945(SplayListForDB)
                                                                       4    0.000    0.000  {method 'execute' of 'sqlite3.Cursor' objects}
{wx._core_.PostEvent}                                         ->
ClientDB.py:1014(GenerateSpecificMappingsCacheTableNames)     ->
ClientDB.py:4621(_GetServiceIds)                              ->       2    0.000    0.000  HydrusData.py:945(SplayListForDB)
                                                                       2    0.000    0.000  {method 'execute' of 'sqlite3.Cursor' objects}
HydrusTags.py:150(CleanTag)                                   ->       1    0.000    0.000  HydrusData.py:1028(ToUnicode)
                                                                       1    0.000    0.000  re.py:138(match)
                                                                       3    0.000    0.000  re.py:148(sub)
                                                                       1    0.000    0.000  {method 'lower' of 'unicode' objects}
                                                                       2    0.000    0.000  {method 'replace' of 'unicode' objects}
HydrusData.py:945(SplayListForDB)                             ->       7    0.000    0.000  {method 'join' of 'str' objects}
HydrusPubSub.py:137(pub)                                      ->       3    0.000    0.000  ClientController.py:699(NotifyPubSubs)
                                                                       3    0.000    0.000  {method 'append' of 'list' objects}
{method 'sub' of '_sre.SRE_Pattern' objects}                  ->
collections.py:458(__init__)                                  ->       2    0.000    0.000  collections.py:528(update)
                                                                       2    0.000    0.000  {len}
threading.py:373(notify)                                      ->       1    0.000    0.000  threading.py:64(_note)
                                                                       1    0.000    0.000  threading.py:300(_is_owned)
                                                                       1    0.000    0.000  {method 'release' of 'thread.lock' objects}
                                                                       1    0.000    0.000  {method 'remove' of 'list' objects}
ClientDB.py:2970(_GetHashIds)                                 ->       1    0.000    0.000  {len}
                                                                       1    0.000    0.000  {method 'add' of 'set' objects}
                                                                       1    0.000    0.000  {method 'execute' of 'sqlite3.Cursor' objects}
                                                                       1    0.000    0.000  {method 'fetchone' of 'sqlite3.Cursor' objects}
_core.py:7583(__init__)                                       ->       3    0.000    0.000  _core.py:7590(_SetSelf)
                                                                       3    0.000    0.000  {wx._core_.PyEvent_swiginit}
                                                                       3    0.000    0.000  {wx._core_.new_PyEvent}
ClientDB.py:1498(_CacheCombinedFilesMappingsUpdate)           ->       1    0.000    0.000  ClientDB.py:996(GenerateCombinedFilesMappingsCacheTableName)
                                                                       3    0.000    0.000  {method 'executemany' of 'sqlite3.Cursor' objects}
HydrusData.py:945(<genexpr>)                                  ->
re.py:230(_compile)                                           ->
ClientDB.py:4058(_GetNamespaceIdTagId)                        ->       1    0.000    0.000  HydrusTags.py:135(CheckTagNotEmpty)
                                                                       1    0.000    0.000  HydrusTags.py:150(CleanTag)
                                                                       2    0.000    0.000  {method 'execute' of 'sqlite3.Cursor' objects}
                                                                       2    0.000    0.000  {method 'fetchone' of 'sqlite3.Cursor' objects}
ClientDB.py:4607(_GetServiceId)                               ->       1    0.000    0.000  {method 'execute' of 'sqlite3.Cursor' objects}
                                                                       1    0.000    0.000  {method 'fetchone' of 'sqlite3.Cursor' objects}
{method 'fetchone' of 'sqlite3.Cursor' objects}               ->
re.py:148(sub)                                                ->       3    0.000    0.000  re.py:230(_compile)
                                                                       3    0.000    0.000  {method 'sub' of '_sre.SRE_Pattern' objects}
HydrusController.py:142(pub)                                  ->       3    0.000    0.000  HydrusPubSub.py:137(pub)
{wx._core_.new_PyEvent}                                       ->
ClientController.py:699(NotifyPubSubs)                        ->       3    0.000    0.000  _core.py:16750(CallAfter)
threading.py:285(__enter__)                                   ->       1    0.000    0.000  {method '__enter__' of 'thread.lock' objects}
{wx._core_.GetApp}                                            ->
threading.py:576(set)                                         ->       1    0.000    0.000  threading.py:285(__enter__)
                                                                       1    0.000    0.000  threading.py:288(__exit__)
                                                                       1    0.000    0.000  threading.py:400(notifyAll)
ClientDB.py:8645(_Write)                                      ->       1    0.000    0.002  ClientDB.py:5519(_ProcessContentUpdates)
ClientDB.py:4510(_GetService)                                 ->       1    0.000    0.000  ClientData.py:1338(GetServiceType)
_core.py:8421(GetApp)                                         ->       3    0.000    0.000  {wx._core_.GetApp}
{len}                                                         ->
HydrusDB.py:317(_GetRowCount)                                 ->
HydrusData.py:1885(PutResult)                                 ->       1    0.000    0.000  threading.py:576(set)
{method 'append' of 'list' objects}                           ->
HydrusData.py:1028(ToUnicode)                                 ->       2    0.000    0.000  {isinstance}
threading.py:300(_is_owned)                                   ->       1    0.000    0.000  {method 'acquire' of 'thread.lock' objects}
threading.py:400(notifyAll)                                   ->       1    0.000    0.000  threading.py:373(notify)
                                                                       1    0.000    0.000  {len}
{wx._core_.PyEvent__SetSelf}                                  ->
{method 'replace' of 'unicode' objects}                       ->
{wx._core_.Event_SetEventType}                                ->
collections.py:528(update)                                    ->       2    0.000    0.000  {len}
HydrusDB.py:528(pub_after_commit)                             ->       3    0.000    0.000  {method 'append' of 'list' objects}
{method 'difference' of 'set' objects}                        ->
ClientDB.py:1000(GenerateMappingsTableNames)                  ->
ClientDB.py:8265(<setcomp>)                                   ->
ClientDB.py:8124(<setcomp>)                                   ->
{method 'match' of '_sre.SRE_Pattern' objects}                ->
re.py:138(match)                                              ->       1    0.000    0.000  re.py:230(_compile)
                                                                       1    0.000    0.000  {method 'match' of '_sre.SRE_Pattern' objects}
<string>:1(<module>)                                          ->       1    0.000    0.164  HydrusDB.py:451(_ProcessJob)
_core.py:7590(_SetSelf)                                       ->       3    0.000    0.000  {wx._core_.PyEvent__SetSelf}
{wx._core_.PyEvent_swiginit}                                  ->
ClientDB.py:1748(<genexpr>)                                   ->
ClientDB.py:8686(pub_content_updates_after_commit)            ->       2    0.000    0.000  HydrusDB.py:528(pub_after_commit)
{method 'update' of 'set' objects}                            ->       2    0.000    0.000  ClientDB.py:8256(<genexpr>)
threading.py:288(__exit__)                                    ->       1    0.000    0.000  {method '__exit__' of 'thread.lock' objects}
{method 'items' of 'dict' objects}                            ->
HydrusTags.py:135(CheckTagNotEmpty)                           ->
ClientDB.py:996(GenerateCombinedFilesMappingsCacheTableName)  ->
{method 'lower' of 'unicode' objects}                         ->
HydrusData.py:1789(ToTuple)                                   ->
_core.py:4972(SetEventType)                                   ->       3    0.000    0.000  {wx._core_.Event_SetEventType}
{method 'intersection' of 'set' objects}                      ->
{method 'add' of 'set' objects}                               ->
_core.py:8403(PostEvent)                                      ->       3    0.000    0.000  {wx._core_.PostEvent}
{isinstance}                                                  ->
{method 'union' of 'set' objects}                             ->
ClientDB.py:8123(<setcomp>)                                   ->
ClientDB.py:8105(<setcomp>)                                   ->
ClientDB.py:8264(<setcomp>)                                   ->
ClientDB.py:8121(<genexpr>)                                   ->
{hasattr}                                                     ->
HydrusData.py:1883(IsSynchronous)                             ->
ClientDB.py:8206(<genexpr>)                                   ->
ClientDB.py:8102(<setcomp>)                                   ->
ClientDB.py:8256(<genexpr>)                                   ->
ClientDB.py:8120(<genexpr>)                                   ->
{built-in method from_iterable}                               ->
ClientDB.py:1504(<genexpr>)                                   ->
HydrusData.py:1858(GetCallableTuple)                          ->
ClientData.py:1338(GetServiceType)                            ->
threading.py:64(_note)                                        ->
ClientDB.py:8255(<genexpr>)                                   ->
ClientDB.py:1506(<genexpr>)                                   ->
HydrusData.py:1881(GetType)                                   ->
ClientDB.py:8112(<setcomp>)                                   ->
{method 'remove' of 'list' objects}                           ->
{callable}                                                    ->
{method '__enter__' of 'thread.lock' objects}                 ->
ClientDB.py:1502(<genexpr>)                                   ->
ClientDB.py:8109(<setcomp>)                                   ->
{method 'acquire' of 'thread.lock' objects}                   ->
ClientDB.py:8208(<setcomp>)                                   ->
collections.py:479(__missing__)                               ->
{method 'release' of 'thread.lock' objects}                   ->
{method '__exit__' of 'thread.lock' objects}                  ->
{method 'disable' of '_lsprof.Profiler' objects}              ->

license: wtfpl


code:
		<title>reducing lag</title>
		<link href="hydrus.ico" rel="shortcut icon" />
		<link href="style.css" rel="stylesheet" type="text/css" />

hydrus is cpu and hdd hungry

The hydrus client manages a lot of complicated data and gives you a lot of power over it. To add millions of files and tags to its database, and then to perform difficult searches over that information, it needs to use a lot of CPU time and hard drive time--sometimes in small laggy blips, and occasionally in big 100% CPU chunks. I don't put training wheels or limiters on the software either, so if you search for 300,000 files, the client will try to fetch that many.

In general, the client works best on snappy computers with low-latency hard drives where it does not have to constantly compete with other CPU- or HDD- heavy programs. Running hydrus on your games computer is no problem at all, but you should have it set to not start a big job while your CPU is otherwise busy so your games can run freely. Similarly, if you run two clients on the same computer, you should have them set to work at different times, because if they both try to process 500,000 tags at once on the same hard drive, they will each slow to a crawl.

Keeping your HDDs defragged is very important, and good practise for all your programs anyway. Make sure you know what this is and that you do it. I use PerfectDisk. O&O Defrag is also good.

maintenance and processing

I have attempted to offload most of the background maintenance of the client (which typically means repository processing and internal database defragging) to time when you are not using the client. This can either be 'idle time' or 'shutdown time'. The calculations for what these exactly mean are customisable in file->options->maintenance and processing.

If you run a quick computer, you likely don't have to change any of these options. Repositories will synchronise and the database will stay fairly optimal without you even noticing the work that is going on. This is especially true if you leave your client on all the time.

If you have an old, slower computer though, or if your hard drive is high latency for one reason or another (e.g. you use encryption), make sure these options are set for whatever is best for your situation. Turning off idle time completely is often helpful as some older computers are slow to even recognise--mid task--that you want to use the client again, or take too long to abandon a big task half way through. If you set your client to only do work on shutdown, then you can control exactly when that happens.

Keeping the database vacuumed is important, so if you remove it from the normal maintenance schedule, make sure you run it every now and then manually from database->maintenance->vacuum. It takes a few minutes to run, but it is great for cleaning up a database recently fragged by several million new rows of data.

reducing search and general gui lag

Searching for tags via the autocomplete dropdown and searching for files in general can sometimes take a very long time. It depends on many things. In general, the more predicates (tags and system:something) you have active for a search, the faster it will be. And the more specific the search domain (e.g. "local files" instead of "all known files" and "my tag repo" instead of "all known tags"), the faster it will be.

You can also look at file->options->speed and memory, again especially if you have a slow computer. Increasing the autocomplete thresholds is very often helpful. You can even force autocompletes to only fetch results when you manually ask for them.

Having lots of thumbnails open can slow many things down. If you get lag with 10,000 files open in your searches, try cutting it down to only 1,000 or so. Split your downloading binges and subscriptions into smaller, rarer chunks, and don't try to watch 1080p webms while five other things are going on.

finally - profiles

Lots of my code remains unoptimised for certain situations. My development environment is obviously specific to me and has only a few thousand images and a few million tags. As I write code, I am usually more concerned with getting it to work at all rather than getting it to work fast for every possible scenario. So, if something is running particularly slow for you, but your computer is otherwise working fine, let me know and I can almost always speed it up.

Let me know:

The general steps to reproduce the problem (e.g. "Running system:numtags>4 is ridiculously slow on its own on 'all known tags'.")

Your operating system and its version (e.g. "Windows 8.1")

Your computer's general power (e.g. "A couple of years old. It runs most stuff ok.")

The type of hard drive you are running hydrus from. (e.g. "A 2TB 7200rpm drive that is 20% full. I regularly defrag it.")

Any profiles you have collected.

A profile is a large block of debug text that lets me know which parts of my code are running slow for you. Currently, hydrus profiles have three sections. A complete one looks like this.

It is very helpful to me to have a profile. You can generate some by going help->debug->db profile mode, which tells the client to generate profile information for every subsequent database request. This will spam your logfile, so don't leave it on for a very long time (you can turn it off by hitting the help menu entry again).

Turn on profile mode, do the thing that runs slow for you (importing a file, fetching some tags, whatever), and then shut the client down and go to the newly created profile logfile, which should be at install_dir/db/client profile - TIMESTAMP.log. This file will be filled with several sets of tables with timing information for one (or likely more) code calls. You can either copy and paste the data labelled for your problem request (e.g. "import_file") or just send me the whole logfile.

pubsub profile mode is experimental and very log heavy. Feel free to play with it, but it is really for my own purposes. Almost everything that is slow in the program is due to my inefficient database queries.

There are several ways to contact me.


code:
		<title>running from source</title>
		<link href="hydrus.ico" rel="shortcut icon" />
		<link href="style.css" rel="stylesheet" type="text/css" />

running from source

I write the client and server entirely in python, which can run straight from source. It is not simple to get hydrus running this way, but if none of the built packages work for you (for instance you use a non-Ubuntu-compatible flavour of Linux), it may be the only way you can get the program to run. Also, if you have a general interest in exploring the code or wish to otherwise modify the program, you will obviously need to do this stuff.

a quick note about Linux flavours

I often point people here when they are running non-Ubuntu flavours of Linux and cannot run my build. One Debian user mentioned that he had an error like this:

ImportError: /home/user/hydrus/libX11.so.6: undefined symbol: xcb_poll_for_reply64

But that by simply deleting the libX11.so.6 file in the hydrus install directory, he was able to boot. I presume this meant my hydrus build was then relying on his local libX11.so, which happened to have better API compatibility. If you receive a similar error, you might like to try the same sort of thing. Let me know if you discover anything!

what you will need

You will need basic python experience, python 3.x and a number of python modules. Most of it you can get through pip.

If you are on Linux or OS X, or if you are on Windows and have an existing python you do not want to stomp all over with new modules, I recommend you create a virtual environment:

Note, trying to get wx working in a virtual environment on OS X can be a mega-pain. TBH, if you want to run the client on OS X, it is a lot easier to just pip-install to your OS X's python (if that isn't a conflict with what else you are doing).

(navigate to your hydrus extract folder)

pip3 install virtualenv (if you need it)

mkdir venv

virtualenv venv

. venv/bin/activate

That '. venv/bin/activate' line turns your venv on, and will be needed every time you run the client.pyw/server.py files. You can easily tuck it into a launch script.

After that, you can go nuts with pip. I think this will do for most systems:

pip3 install beautifulsoup4 chardet html5lib lxml nose numpy opencv-python six Pillow psutil PyOpenSSL PyYAML requests Send2Trash service_identity twisted

You may want to do all that in smaller batches.

And optionally, you can add these packages:

lz4 - for some memory compression in the client

pylzma - for importing rare ZWS swf files

pysocks - for socks4/socks5 proxy support (although you may want to try "requests[socks]" instead)

matplotlib - for some bandwidth graph drawing

mock httmock pyinstaller - if you want to run test.py and make a build yourself

Depending on your OS, you might need something else. Ultimately, the best way to figure out if you have enough for hydrus is to just keep running client.pyw and see what it complains about missing. If you are not familiar with pip and get an error about a library already existing, know that you update an existing library with the --upgrade switch, like so:

pip3 install --upgrade libraryname

For Windows, depending on which compiler you are using, pip can have problems building some modules like lz4 and lxml. This page has a lot of prebuilt binaries--I have found it very helpful many times. You may want to update python's sqlite3.dll as well--you can get it here, and just drop it in C:\Python27\DLLs or wherever you have python installed. I have a fair bit of experience with Windows python, so send me a mail if you need help.

You will also need wxPython 4.0. This can be simple or complicated. For Windows and OS X, just get it with pip like the rest:

pip3 install wxPython

If you run Linux, getting wxPython from pip needs a build, which in my experience is a time-consuming and frequently failing pain. Here is their page on it.

They have some wheels for common distros here, which I highly recommend trying before you attempt to build. Even if your distro is not listed, try one that looks similar. On Ubuntu 16.04, I had more luck with the gtk2 build. You'll be running a command like:

pip3 install -f https://extras.wxpython.org/wxPython4/extras/linux/gtk2/ubuntu-16.04 wxPython

Even with a wheel, you might need some additional dependencies. In my different attempts to build or just get the wheel going, I needed one or more of these:

sudo apt-get install libsdl1.2-dev libwebkitgtk3.0 libwebkitgtk3.0-dev gstreamer-1.0 libgstreamer-plugins-base1.0-dev v4l-utils freeglut3-dev libgtk-3-dev libglw1-mesa libglw1-mesa-dev python-dev

I think the wheel might only need libsdl1.2-dev on Ubuntu 16.04, but your situation may be different. Again, this was a pain to figure out, so good luck!

YMMV. Feel free to email me if you run into trouble or discover any neat tricks.

OS X 10.9 is easier these days. If you can't get opencv working from pip, this should do it:

brew tap homebrew/homebrew-science

brew install opencv (or try opencv3, although I have had trouble with this)

export PYTHONPATH=/usr/local/lib/python2.7/site-packages:$PYTHONPATH

If you don't have ffmpeg in your path and you want to import videos, you will need to put a static FFMPEG executable in the install_dir/bin directory. Have a look at how I do it in the extractable compiled releases if you can't figure it out. You can either copy the exe from one of those releases, or download the latest build right from the FFMPEG site. I don't include these exes in the source release just because they are so big.

Once you have everything set up, client.pyw and server.py should look for and run off client.db and server.db just like the executables. They will look in the 'db' directory by default, or anywhere you point them with the "-d" parameter.

I develop hydrus on and am most experienced with Windows, so the program is much more stable and reasonable on that. I do not have as much experience with Linux or OS X, so I would particularly appreciate your Linux/OS X bug reports and any informed suggestions.

my code

Unlike most software people, I am more INFJ than INTP/J. My coding style is unusual and unprofessional, and everything is pretty much hacked together. Please look through the source if you are interested in how things work and ask me if you don't understand something. I'm constantly throwing new code together and then cleaning and overhauling it down the line.

I work alone, so while I am very interested in detailed bug reports or suggestions for good libraries to use, I am not looking for pull requests. Everything is WTFPL, so feel free to fork and play around with things on your end as much as you like.


code:
		<title>running your own server</title>
		<link href="hydrus.ico" rel="shortcut icon" />
		<link href="style.css" rel="stylesheet" type="text/css" />

The server is hacked-together and quite technical. It requires a fair amount of experience with the client and its concepts, and it does not operate on a timescale that works well on a LAN. I suggest you only try running your own server once you have a bit of experience synchronising with mine or someone else's and you think, 'Hey, I know exactly what that does, and I would like one!'

setting up a server

I will use two terms, server and service, to mean two distinct things:

A server is an instantiation of the hydrus server executable (e.g. server.exe in Windows). It has a complicated and flexible database that can run many different services in parallel.

A service sits on a port (e.g. 45871) and responds to certain http requests (e.g. /file or /update) that the hydrus client can plug into. A service might be a repository for a certain kind of data, the administration interface to manage what services run on a server, or anything else.

Setting up a hydrus server is easy compared to, say, Apache. There are no .conf files to mess about with, and everything is controlled through the client. When started, the server will place an icon in your system tray in Windows or open a small frame in Linux or OS X. To close the server, either right-click the system tray icon and select exit, or just close the frame.

The basic process for setting up a server is:

Start the server.

Set up your client with its address and initialise the admin account

Set the server's options and services.

Make some accounts for your users.

???

Profit

Let's look at these steps in more detail:

start the server

Since the server and client have so much common code, I package them together. If you have the client, you have the server. If you installed in Windows, you can hit the shortcut in your start menu. Otherwise, go straight to 'server' or 'server.exe' or 'server.pyw' in your installation directory. The program will first try to take port 45870 for its administration interface, so make sure that is free. Open your firewall as appropriate.

set up the client

In the services->manage services dialog, add a new 'hydrus server administration service' and set up the basic options as appropriate. If you are running the server on the same computer as the client, its hostname is 'localhost'.

In order to set up the first admin account and an access key, use 'init' as a registration key. This special registration key will only work to initialise this first super-account.

YOU'LL WANT TO SAVE YOUR ACCESS KEY IN A SAFE PLACE

If you lose your admin access key, there is no way to get it back, and if you are not sqlite-proficient, you'll have to restart from the beginning by deleting your server's database files.

If the client can't connect to the server, it is either not running or you have a firewall/port-mapping problem. If you want a quick way to test the server's visibility, just put https://host:port into your browser (make sure it is https! http will not work)--if it is working, your browser will probably complain about its self-signed https certificate. Once you add a certificate exception, the server should return some simple html identifying itself.

set up the server

You should have a new submenu, 'administrate services', under 'services', in the client gui. This is where you control most server and service-wide stuff.

admin->your server->manage services lets you add, edit, and delete the services your server runs. Every time you add one, you will also be added as that service's first administrator, and the admin menu will gain a new entry for it.

making accounts

Go admin->your service->create new accounts to create new registration keys. Send the registration keys to the users you want to give these new accounts. A registration key will only work once, so if you want to give several people the same account, they will have to share the access key amongst themselves once one of them has registered the account. (Or you can register the account yourself and send them all the same access key. Do what you like!)

Go admin->manage account types to add, remove, or edit account types. Make sure everyone has at least downloader (get_data) permissions so they can stay synchronised.

You can create as many accounts of whatever kind you like. Depending on your usage scenario, you may want to have all uploaders, one uploader and many downloaders, or just a single administrator. There are many combinations.

???

The most important part is to have fun! There are no losers on the INFORMATION SUPERHIGHWAY.

profit

I honestly hope you can get some benefit out of my code, whether just as a backup or as part of a far more complex system. Please mail me your comments as I am always keen to make improvements.

btw, how to backup a repo's db

All of a server's files and options are stored in its accompanying .db file and respective subdirectories, which are created on first startup (just like with the client). To backup or restore, you have two options:

Shut down the server, copy the database files and directories, then restart it. This is the only way, currently, to restore a db.

In the client, hit admin->your server->make a backup. This will lock the db server-side while it makes a copy of everything server-related to server_install_dir/db/server_backup. When the operation is complete, you can ftp/batch-copy/whatever the server_backup folder wherever you like.

OMG EVERYTHING WENT WRONG

If you get to a point where you can no longer boot the repository, try running SQLite Studio and opening server.db. If the issue is simple--like manually changing the port number--you may be in luck. Send me an email if it is tricky.

Remember that everything is breaking all the time. Make regular backups, and you'll minimise your problems.


code:
a { color: #222; text-decoration: none; font-weight: bold; }
a:hover { color: #555 }
body { font-family: "Calibri", Arial, sans-serif; color: #555; line-height: 1.5; }
h3 { color: #222; }
ul li { list-style: none; margin: 1.0em 0em; }
ul.bulletpoint li { list-style: disc; margin: 1.0em 0em; }
th { text-align: center; }
tr { text-align: center; }

.de { color: #d00 }
.su { color: #0d0 }
.dealwithit { color: #ff0000; text-shadow:
1px 1px 0 #ff7f00,
2px 2px 0 #ffff00,
3px 3px 0 #00ff00,
4px 4px 0 #0000ff,
5px 5px 0 #6600ff,
6px 6px 0 #8b00ff; }
.warning { color: #d00 }
.apiborder { border-left: 5px; border-left-color: #333377; border-style: hidden hidden hidden solid; padding: 0.5em; margin: 1.0em; }
.lololol { text-decoration: line-through; }
.right { text-align: right; }
.screenshot { float: right; clear: both; margin: 10px; }
.spoiler:not(:hover) { color: #000; background-color: #000 }


code:
		<title>support</title>
		<link href="hydrus.ico" rel="shortcut icon" />
		<link href="style.css" rel="stylesheet" type="text/css" />

can I contribute to hydrus development?

I don't expect anything! I'm amazed and grateful that anyone wants to use my software or upload to the public tag repository. I enjoy the feedback and work, and I hope to keep putting completely free weekly releases out as long as there is more to do.

That said, as I have developed the software, several users have kindly offered to contribute money, either as thanks for a specific feature or just in general. I kept putting the thought off, but I eventually got over my hesitance and set something up.

I find the tactics of most internet fundraising very distasteful, especially when they promise something they then fail to deliver.  I much prefer the 'if you like me and would like to contribute, then please do, meanwhile I'll keep doing what I do' model. I support Red Letter Media and a few other 'put out regular free content' creators on Patreon in this way, and I get a lot out of it, even though I have no direct reward beyond the knowledge that I helped some people do something neat.

If you feel the same way about my work, I've set up a simple Patreon page here.