Organizing electronic references

[geotechguy1]

Can't think of a good place to make this post.

Just curious how others manage electronic references. I've accumulated thousands of journal articles, technical manuals, textbooks, example projects over the years and I'm finding that managing it is getting difficult. I've tried various folder based systems but eventually it breaks down (and also this system usually requires having a USB key or hard drive to shuttle back and forth between home and work which employers view with suspicion now).

Just wondering if anyone knows of any software or tools to organize electronically effectively.

 

[CWB1]

The only thing I ever do is ensure that documents are not simply scanned images, but searchable data. Adobe and others have tools for automatically converting images to text if needed. So long as the docs are searchable then a standard windows search should be able to find relevant keywords within documents in a particular folder or drive.

Disclaimer: Various digital libraries that I use for work and entertainment usually sort docs into various digital bins. I do find that useful for choosing random books for education or entertainment, but keyword searches like described above are much more useful when I have a small inkling of what I'm looking for.

 

[skeletron]

Yes, this can be an issue and I don't know if there is a really efficient way to organize it other than being diligent to a system and following it.

For ease of access: use some sort of Cloud service. Google Drive or MS OneDrive, etc. I think Google would at least have good search capabilities.

For ease of retrieval: use the meta data in the file, or as @CWB1 suggests, use OCR when scanning. I have good success using Windows search and a few key words. Most of mine are a hodge-podge of scanned, digital, and older references with no extra meta data. I use folders and names as the main targets because I still rely on my muscle memory at times.

For both: engage a Content Management System. This seems overkill for an employee, but it could really do all that you want. Basically, it would be a cloud type service but with enhanced search capabilities and you can also "tag" and "link" documents. It's similar to how bloggers group their posts and are able to link their content.

 

[milkshakelake]

About shuttling a USB key or hard drive back and forth: besides cloud services, you can also use something like Syncthing (free) to sync it between a company server and home server (if that kind of thing is allowed). I use both - Onedrive cloud for backup, Syncthing for everyday syncing.

One thing that helps me with folder management is renaming example project files with some key words. For example, instead of naming a project "1000 Engtips Ave", I'd name it "1000 Engtips Ave flat slab tiebacks piles" or "1000 Engtips Ave plank CMU steel". I haven't tried OCR with example projects but Windows search will pick up the useful words in the filename.

 

[pmover]

i sorted efiles by discipline (mech, elec, civil, structural, inst., etc), then by organization (i.e. API, ASME, NEC, ASCE, IEC, etc.), and then by subject matter. I carried a flash drive until flash drives were no longer permitted by company IT. if i needed data from a paper or otherwise, i extracted the page and emailed to work address. never use cloud services. as CWB stated, if possible, creating searchable text files via OCR was done.
now that im retired, i have been deleting these files.

 

[xnuke]

I currently use Calibre available here for library management. It's running as a portable app on a portable hard drive. My IT department is going to lock down all USB ports so no mass storage devices can connect, so I'll have to access my library from a personal computer or I'll have to put it into a cloud location. I'm not sure which I'll choose.

xnuke
"Live and act within the limit of your knowledge and keep expanding it to the limit of your life." Ayn Rand, Atlas Shrugged.
Please see FAQ731-376 for tips on how to make the best use of Eng-Tips.

 

[Sparweb]

It's a common problem, especially for information pack-rats like me.

I started a thread a few years back about this. There's software that you can use to help sort and search. Ultimately, that thread also revealed to me that I was not using Windows indexing service for my hard drives, and that solved the problem on my home computer. Today it stands at 204GB. Nearly 400k files!

https://www.eng-tips.com/viewthread.cfm?qid=436979

To make it convenient to routinely look things up, I organize my files like others have suggested, by broad subject and engineering discipline, with subfolders as appropriate from there.

Aircraft
Automotive
CAD
Catalogs
CFD
Chemical
Civil
Culture
Drafting+Drawing+Design
Electrical
Fasteners
Materials
Math+Science_Ref
Mech_Eng
Programming
Software
Standards+Specs
Textbooks

To deploy that solution at my workplace, however, was a different matter. Our IT department is not keen on having network server files indexed and searchable, because it can enable data theft, should a break-in ever happen. I still haven't solved that problem, but having a huge repository of information at home helps (larger than the one at work, in fact).

 

[drawoh]

geotechguy1,

I keep HTML files on my appropriate computers. HTML is easy to code with a text editor. It is easy to create navigation in a long HTML file. Links to other web pages is what HTML is all about. The definition lists are an awesome tool for listing vendors and contact.

HTML is idiot simple to write, almost certainly simpler than the applications that help you write HTML.

--
JHG

 

[IRstuff]

Our company blocks all online storage sites, so the only thing I've got is a company approved external drive. Nevertheless, most categorization techniques will eventually break down; that's why the Dewey Decimal system was invented.

SW, your files must be more efficient than mine; I've got about 400k files, but they occupy 433GB.

TTFN (ta ta for now)
I can do absolutely anything. I'm an expert!

https://www.youtube.com/watch?v=BKorP55Aqvg

faq731-376 forum1529 Entire Forum list

http://www.eng-tips.com/forumlist.cfm

 

[Sparweb]

Or they're just longer. I may have a shorter attention span than you.

 

[Ron247]

It would appear you do need to create a filing system again. Sounds like you have once at least, but I see no other way to store the amount of data people note here. Problem with any system is how fast it changes when you start using it. That is when I always see something I did not think out. In Spars example above, I see catalogs, software, mechanical, and electrical in the main categories. I had something similar but about has fast as I started using it, I have to take catalogs for example and then subdivide it into electrical, mech etc. As first I did not, at that time, there were not that many pdf catalogs. I think I had 5 or 6. So, no need to subdivide. Then, boom the world of pdf created hundreds of catalogs. I go back to the 8 character in a filename days. Not a lot of computer literature, just programs and text files.

The topics on one level need a common theme. That needs to also have misc, undecided and archive included. Next level similar. Level 1 Arch, MEch, Ele etc. Level 2; softare, catalogs, programs etc. with misc, undecided, and archive. Whatever fits what you do. Once you create those 2nd and 3rd level directories, you can cut and paste them into any new Level 1 you create. Don't have to type them every time.

If you need a quick way to get a text file that contains all your existing files and where they are at, that can easily be done with a Command Line.

 

[IRstuff]

I've tried folder hierarchies, but Windoze has a limitation on the total number of characters in a pathname, down to the filename, so if you want descriptive foldernames and filenames, you're in a bind, like me. I had to drop the top two levels of categories, leaving me with around 6000 subfolders in my publications folder.

I've thought about using some sort of database program to manage that, but that just seems to be an extra layer of complexity, so haven't pursued it much

TTFN (ta ta for now)
I can do absolutely anything. I'm an expert!

https://www.youtube.com/watch?v=BKorP55Aqvg

faq731-376 forum1529 Entire Forum list

http://www.eng-tips.com/forumlist.cfm

 

[Ron247]

Is Windoze something I am not familiar with or is it Windows spelled different? I know I have seen tree structures with a lot of characters. I thought the path and filename was at least 200.

 

[IRstuff]

The default limit in Windoze (Windows) is around 255, which derives from its Windows NT roots. Supposedly, in Windows 10, they allowed for much longer path/file names, but it's not easy to change and I get errors trying to do so on my machine. Nevertheless, the default limit is often breached, for me, at least; I've hit the limit at least a half-dozen times, and oddly, it mostly manifests when I try to delete files, so there's possibly something different when different functions are used.

TTFN (ta ta for now)
I can do absolutely anything. I'm an expert!

https://www.youtube.com/watch?v=BKorP55Aqvg

faq731-376 forum1529 Entire Forum list

http://www.eng-tips.com/forumlist.cfm

 

[dik]

For technical stuff... I use a Masterformat approach. I use it less and less, relying on more current information, available.

Rather than think climate change and the corona virus as science, think of it as the wrath of God. Feel any better?

-Dik

 

[IRstuff]

Yeah, I've got WAAAY more than 50 categories. ;-) more like 740 top level folders. Of course, I claim no rational organization, it's actually very ad hoc; some things are clearly at the wrong level and others have too many sublevels or not enough. At this point, it's also A LOT of work needed to get them all in the "right" places, so meh...

What's most important is the filenames; don't skimp on characters here and use a good search tool. I use Void Tools EVERYTHING, which seems much faster than Explorer's search engine, at least as far as filenames are concerned.

TTFN (ta ta for now)
I can do absolutely anything. I'm an expert!

https://www.youtube.com/watch?v=BKorP55Aqvg

faq731-376 forum1529 Entire Forum list

http://www.eng-tips.com/forumlist.cfm

 

[dik]

The files can quickly become 'nested' and some of them seem to fit in more than directory. Just recently posted a retaining wall program on the site... couldn't find it, but was looking for something else and came across it. When you have half a meg in files... it sometimes becomes a challenge.

Rather than think climate change and the corona virus as science, think of it as the wrath of God. Feel any better?

-Dik

 

[IRstuff]

Sure, when I look for a particular file, I sometimes find 12 or more copies of it. I tend to have multiple backups to start with and, as you say, they wind up being copied into a current project's folder to make it easier to find during the course of the project. But, there's usually no time to clean up after project's completed, leaving breadcrumbs all over the place; it's not a pretty sight. Then, there's the no so pretty memory, so a file might get copied into multiple folders within a project. Good thing retirement is on the near horizon ;-)

TTFN (ta ta for now)
I can do absolutely anything. I'm an expert!

https://www.youtube.com/watch?v=BKorP55Aqvg

faq731-376 forum1529 Entire Forum list

http://www.eng-tips.com/forumlist.cfm

 

[Ron247]

IRStuff,
I have also ran into routines or programs that do not allocate enough room to contain the full path and filename.

 

[geotechguy1]

I've ended up moving most of my reference material into a program called Mendely:

https://www.mendeley.com/

It took a bit to get set up but once things are already in there adding new references is a piece of piss