diff --git a/.github/workflows/regular-tests.yaml b/.github/workflows/regular-tests.yaml index 148f3f0f..fbcfb625 100644 --- a/.github/workflows/regular-tests.yaml +++ b/.github/workflows/regular-tests.yaml @@ -21,7 +21,7 @@ jobs: - uses: actions/checkout@v4 - uses: actions/setup-go@v5 with: - go-version: '1.23.3' + go-version: 'stable' - uses: golangci/golangci-lint-action@v5 with: version: v1.60 diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml index d33db5a9..cfd413c5 100644 --- a/.github/workflows/release.yaml +++ b/.github/workflows/release.yaml @@ -23,7 +23,7 @@ jobs: id: extract_branch - uses: actions/setup-go@v5 with: - go-version: 1.22.2 + go-version: 'stable' - name: Create Release uses: softprops/action-gh-release@v2 with: diff --git a/.gitignore b/.gitignore index 9162da20..4c6db7aa 100644 --- a/.gitignore +++ b/.gitignore @@ -14,6 +14,7 @@ rice-box.go /backend/test_config.yaml /backend/srv /backend/http/dist +/backend/http/embed/* .DS_Store node_modules diff --git a/CHANGELOG.md b/CHANGELOG.md index 49e4fcc4..6dc4faa3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,12 +2,50 @@ All notable changes to this project will be documented in this file. For commit guidelines, please refer to [Standard Version](https://github.com/conventional-changelog/standard-version). +## v0.3.5 + + **New Features** + - More indexing configuration options possible. However consider waiting on using this feature, because I will soon have a full onboarding experience in the UI to manage sources instead. + - added config file options "sources" in the server config. + - can enable/disable indexing a specified list of directories/files + - can enable/disable indexing hidden files + - prepped for multiple sources (not supported yet!) + - Theme and Branding support (see updates to [configuration wiki](https://github.com/gtsteffaniak/filebrowser/wiki/Configuration) on how to use) + - Automatically expire shares https://github.com/gtsteffaniak/filebrowser/issues/208 + + **Notes**: + - MacOS application files (ending in ".app") were previously treated as folders, now they are treated as a single file. + - No longer indexes "$RECYCLE.BIN" or "System Volume Information" directories. + - Icon styling tweaked so all icons have a background. + - Updated Login page styling. + - Settings profile menu has been simplified, password changes happen in user management. + + **Bugfixes**: + - Fixed setting share expiration time would not work due to type conversion error. + - More safari fixes related to text-selection. + - Sort by name value sorting ignores the extension, only sorts by name https://github.com/gtsteffaniak/filebrowser/issues/230 + - Fixed manual language selection issue. + - Fixed exact date time issue. + + +New login page: + +image + +Example branding in sidebar: + +image2 + +Example user settings page: + +image3 + ## v0.3.4 **Bugfixes**: - Safari right-click actions. - Some small image viewer behavior - - Progressive webapp "install to homescreen" fix. + - Progressive webapp "install to homescreen" fix. ## v0.3.3 diff --git a/README.md b/README.md index a44d8c1d..9b61d94a 100644 --- a/README.md +++ b/README.md @@ -6,32 +6,30 @@

FileBrowser Quantum - A modern web-based file manager

- +

> [!Note] > Starting with v0.3.3, configuration file mapping is different to support non-root user. Now, the default config file name is `config.yaml` and in docker the path is `/home/filebrowser/config.yaml` and `/home/filebrowser/`. Please read the usage below to properly update your config to point the new config location. (open an issue for any help needed) > [!WARNING] -> - There is no stable version yet. Always check release notes for bug fixes on functionality that may have been changed. If you notice any unexpected behavior -- please open an issue to have it fixed soon. +> There is no stable version yet. Always check release notes for bug fixes on functionality that may have been changed. If you notice any unexpected behavior -- please open an issue to have it fixed soon. FileBrowser Quantum is a fork of the file browser opensource project with the following changes: - 1. [x] Indexes files efficiently. (See [indexing readme](./docs/indexing.md) for more info.) - - Real-time search results as you type! + 1. [x] Indexes files efficiently. (See [indexing Wiki](https://github.com/gtsteffaniak/filebrowser/wiki/Indexing) for more info.) + - Real-time search results as you type - Search supports file/folder sizes and many file type filters. - Enhanced interactive results that show file/folder sizes. 2. [x] Revamped and simplified GUI navbar and sidebar menu. - - Additional compact view mode as well as refreshed view mode - styles. + - Additional compact view mode as well as refreshed view mode styles. - Many graphical and user experience improvements. - right-click context menu 3. [x] Revamped and simplified configuration via `config.yaml` config file. 4. [x] Better listing browsing - - Switching view modes is instant - - Folder sizes are shown as well - - Changing Sort order is instant - - The entire directory is loaded in 1/3 the time + - Instantly Switches view modes and sort order without reloading data. + - Folder sizes are displayed + - Navigating remembers the scroll position, navigating back keeps the last scroll position. 5. [x] Developer API support - Ability to create long-live API Tokens. - Helpful Swagger page available at `/swagger` endpoint. @@ -40,9 +38,8 @@ Notable features that this fork *does not* have (removed): - jobs/runners are not supported yet (planned). - shell commands are completely removed and will not be returned. - - Themes and branding are not fully supported yet (planned). + - pagination for directory items for extremely large directories. - see feature matrix below for more. - - pagination for directory items, so large directories with more than 100,000 items may be slow to load or not load at all. ## About @@ -66,7 +63,7 @@ focus of this fork is on a few key principles: - Minimize external dependencies and standard library usage. - Of course -- adding much-needed features. -For more questions, see the [Q&A Readme](./docs/questions.md) +For more, see the [Q&A Wiki](https://github.com/gtsteffaniak/filebrowser/wiki/Q&A) ## Look @@ -84,117 +81,30 @@ a popup menu.

- +

-## Install +## Install and Configuration -Using docker: - -1. docker run (no persistent db): - -``` -docker run -it -v /path/to/folder:/srv -v $(pwd)/config.yaml:/home/filebrowser/config.yaml -p 80:80 gtstef/filebrowser -``` - -or optionally, as non-root filebrowser user: - -``` -docker run -u filebrowser -it -v $(pwd)/config.yaml:/home/filebrowser/config.yaml -v /path/to/folder:/srv -p 80:80 gtstef/filebrowser -``` - -1. docker compose: - - - with local storage - -``` -services: - filebrowser: - volumes: - - '/path/to/folder:/srv' # required (for now not configurable) - # optional if you want db to persist - configure a path under "database" dir in config file. - - './database:/home/filebrowser/database' - - './config.yaml:/home/filebrowser/config.yaml' - ports: - - '80:80' - image: gtstef/filebrowser - # optionally run as non-root filebrowser user - #user: filebrowser - restart: always -``` - - - with network share - -``` -services: - filebrowser: - volumes: - - 'storage:/srv' # required (for now not configurable) - # optional if you want db to persist - configure a path under "database" dir in config file. - - './database:/home/filebrowser/database' - - './config.yaml:/home/filebrowser/config.yaml' - ports: - - '80:80' - image: gtstef/filebrowser - restart: always -volumes: - storage: - driver_opts: - type: cifs - o: "username=admin,password=password,rw" # enter valid info here - device: "//192.168.1.100/share/" # enter valid info here - -``` - -Not using docker (not recommended), download your binary from releases and run with your custom config file: - -``` -./filebrowser -c -``` +See the [Configuration Wiki](https://github.com/gtsteffaniak/filebrowser/wiki/Configuration) ## Command Line Usage -There are very few commands available. There are 3 actions done via the command line: - -1. Running the program, as shown in the install step. The only argument used is the config file if you choose to override the default "config.yaml" -2. Checking the version info via `./filebrowser version` -3. Updating the DB, which currently only supports adding users via `./filebrowser set -u username,password [-a] [-s "example/scope"]` +See the [CLI Wiki](https://github.com/gtsteffaniak/filebrowser/wiki/CLI) ## API Usage -API tokens can be created to perform actions, access file information, and update user settings just like what can be done from the UI. You can create API tokens from the settings page via "API Management" section. This section will only show up if the user has "API" permissions, which can be granted by editing the user in user management. - -Regardless of whether a user has API permissions, anyone can visit the swagger page which is found at `/swagger`. This swagger page uses a short-live token (2-hour exp) that the UI uses, but allows for quick access to all the API's and their described usage and requirements: - -![image](https://github.com/user-attachments/assets/12abd1f6-21d3-4437-98ed-9b0da6cf2c73) - -When using the API outside of swagger, you will need to set the API token as a bearer token authentication type. This means the authorization header will look like `Authorization: Bearer `. For example in Postman: - -Successful Request: - -

image

- -Failed Request - -

image

- +See the [API Wiki](https://github.com/gtsteffaniak/filebrowser/wiki/API) ## Configuration -All configuration is now done via a single configuration file: -`config.yaml`, here is an example of minimal [configuration -file](./backend/config.yaml). - -View the [Configuration Help Page](./docs/configuration.md) for available -configuration options and other help. +Configuration is done via the `config.yaml`, see the [Configuration Wiki](https://github.com/gtsteffaniak/filebrowser/wiki/Configuration) for available configuration options and other help. ## Migration from the original filebrowser -I would recommend that you start fresh without reusing the database. However, -If you want to migrate your existing database to FileBrowser Quantum, -visit the [migration -readme](./docs/migration.md) +See the [Migration +Wiki](https://github.com/gtsteffaniak/filebrowser/wiki/Migration) ## Comparison Chart @@ -246,7 +156,3 @@ Starred/pinned files | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | Content preview icons | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | Plugins support | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | Chromecast support | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | - -## Roadmap - -see [Roadmap Page](./docs/roadmap.md) diff --git a/backend/auth/json.go b/backend/auth/json.go index 54e23d9b..1c2afe07 100644 --- a/backend/auth/json.go +++ b/backend/auth/json.go @@ -7,7 +7,7 @@ import ( "os" "strings" - "github.com/gtsteffaniak/filebrowser/backend/settings" + "github.com/gtsteffaniak/filebrowser/backend/files" "github.com/gtsteffaniak/filebrowser/backend/users" ) @@ -24,7 +24,6 @@ type JSONAuth struct { // Auth authenticates the user via a json in content body. func (a JSONAuth) Auth(r *http.Request, userStore *users.Storage) (*users.User, error) { - config := &settings.Config var cred jsonCred if r.Body == nil { @@ -47,7 +46,7 @@ func (a JSONAuth) Auth(r *http.Request, userStore *users.Storage) (*users.User, return nil, os.ErrPermission } } - u, err := userStore.Get(config.Server.Root, cred.Username) + u, err := userStore.Get(files.RootPaths["default"], cred.Username) if err != nil || !users.CheckPwd(cred.Password, u.Password) { return nil, os.ErrPermission } diff --git a/backend/auth/none.go b/backend/auth/none.go index ca4aeb4b..ab43fc86 100644 --- a/backend/auth/none.go +++ b/backend/auth/none.go @@ -3,7 +3,7 @@ package auth import ( "net/http" - "github.com/gtsteffaniak/filebrowser/backend/settings" + "github.com/gtsteffaniak/filebrowser/backend/files" "github.com/gtsteffaniak/filebrowser/backend/users" ) @@ -15,7 +15,7 @@ type NoAuth struct{} // Auth uses authenticates user 1. func (a NoAuth) Auth(r *http.Request, usr *users.Storage) (*users.User, error) { - return usr.Get(settings.Config.Server.Root, uint(1)) + return usr.Get(files.RootPaths["default"], uint(1)) } // LoginPage tells that no auth doesn't require a login page. diff --git a/backend/auth/proxy.go b/backend/auth/proxy.go index 86183a1e..8a318ab1 100644 --- a/backend/auth/proxy.go +++ b/backend/auth/proxy.go @@ -4,9 +4,8 @@ import ( "net/http" "os" - "github.com/gtsteffaniak/filebrowser/backend/settings" - "github.com/gtsteffaniak/filebrowser/backend/errors" + "github.com/gtsteffaniak/filebrowser/backend/files" "github.com/gtsteffaniak/filebrowser/backend/users" ) @@ -21,7 +20,7 @@ type ProxyAuth struct { // Auth authenticates the user via an HTTP header. func (a ProxyAuth) Auth(r *http.Request, usr *users.Storage) (*users.User, error) { username := r.Header.Get(a.Header) - user, err := usr.Get(settings.Config.Server.Root, username) + user, err := usr.Get(files.RootPaths["default"], username) if err == errors.ErrNotExist { return nil, os.ErrPermission } diff --git a/backend/benchmark_results.txt b/backend/benchmark_results.txt index edfc14f3..06820fa1 100644 --- a/backend/benchmark_results.txt +++ b/backend/benchmark_results.txt @@ -1,43 +1,66 @@ == Running benchmark == /usr/local/go/bin/go -? github.com/gtsteffaniak/filebrowser [no test files] -? github.com/gtsteffaniak/filebrowser/auth [no test files] -? github.com/gtsteffaniak/filebrowser/cmd [no test files] +? github.com/gtsteffaniak/filebrowser/backend [no test files] +? github.com/gtsteffaniak/filebrowser/backend/auth [no test files] +? github.com/gtsteffaniak/filebrowser/backend/cmd [no test files] PASS -ok github.com/gtsteffaniak/filebrowser/diskcache 0.004s -? github.com/gtsteffaniak/filebrowser/errors [no test files] -2024/10/07 12:46:34 could not update unknown type: unknown +ok github.com/gtsteffaniak/filebrowser/backend/diskcache 0.005s +? github.com/gtsteffaniak/filebrowser/backend/errors [no test files] +/home/graham/git/filebrowser /home/graham/git/filebrowser +/home/graham/git/filebrowser/files/file.go /home/graham/git/filebrowser +/home/graham/git/filebrowser/mnt/doesnt/exist /home/graham/git/filebrowser +2025/01/04 14:04:55 Initializing index and assessing file system complexity +2025/01/04 14:04:55 Starting full scan +2025/01/04 14:04:55 Index assessment : complexity=simple directories=0 files=0 +2025/01/04 14:04:55 Time Spent Indexing : 0 seconds +2025/01/04 14:04:55 Next scan in 1m0s +2025/01/04 14:04:56 Initializing index and assessing file system complexity +2025/01/04 14:04:56 Starting full scan +2025/01/04 14:04:56 Index assessment : complexity=simple directories=0 files=0 +2025/01/04 14:04:56 Time Spent Indexing : 0 seconds +2025/01/04 14:04:56 Next scan in 1m0s goos: linux goarch: amd64 -pkg: github.com/gtsteffaniak/filebrowser/files +pkg: github.com/gtsteffaniak/filebrowser/backend/files cpu: 11th Gen Intel(R) Core(TM) i5-11320H @ 3.20GHz -BenchmarkFillIndex-8 10 3847878 ns/op 758424 B/op 5567 allocs/op -BenchmarkSearchAllIndexes-8 10 780431 ns/op 173444 B/op 2014 allocs/op +BenchmarkFillIndex-8 2025/01/04 14:04:57 Initializing index and assessing file system complexity +2025/01/04 14:04:57 Starting full scan +2025/01/04 14:04:57 Index assessment : complexity=simple directories=0 files=0 +2025/01/04 14:04:57 Time Spent Indexing : 0 seconds +2025/01/04 14:04:57 Next scan in 1m0s + 10 3515090 ns/op 34273 B/op 451 allocs/op +BenchmarkCheckIndexExclude-8 10 156.2 ns/op 0 B/op 0 allocs/op +BenchmarkCheckIndexConditionsInclude-8 10 98.00 ns/op 0 B/op 0 allocs/op +2025/01/04 14:04:58 Initializing index and assessing file system complexity +2025/01/04 14:04:58 Starting full scan +2025/01/04 14:04:58 Index assessment : complexity=simple directories=0 files=0 +2025/01/04 14:04:58 Time Spent Indexing : 0 seconds +2025/01/04 14:04:58 Next scan in 1m0s +BenchmarkSearchAllIndexes-8 2025/01/04 14:04:59 Initializing index and assessing file system complexity +2025/01/04 14:04:59 Starting full scan +2025/01/04 14:04:59 Index assessment : complexity=simple directories=0 files=0 +2025/01/04 14:04:59 Time Spent Indexing : 0 seconds +2025/01/04 14:04:59 Next scan in 1m0s + 10 766822 ns/op 34230 B/op 900 allocs/op PASS -ok github.com/gtsteffaniak/filebrowser/files 0.073s +ok github.com/gtsteffaniak/filebrowser/backend/files 5.094s PASS -ok github.com/gtsteffaniak/filebrowser/fileutils 0.003s -2024/10/07 12:46:34 h: 401 -2024/10/07 12:46:34 h: 401 -2024/10/07 12:46:34 h: 401 -2024/10/07 12:46:34 h: 401 -2024/10/07 12:46:34 h: 401 -2024/10/07 12:46:34 h: 401 +ok github.com/gtsteffaniak/filebrowser/backend/fileutils 0.002s PASS -ok github.com/gtsteffaniak/filebrowser/http 0.080s +ok github.com/gtsteffaniak/filebrowser/backend/http 0.184s PASS -ok github.com/gtsteffaniak/filebrowser/img 0.137s +ok github.com/gtsteffaniak/filebrowser/backend/img 0.123s PASS -ok github.com/gtsteffaniak/filebrowser/rules 0.002s +ok github.com/gtsteffaniak/filebrowser/backend/runner 0.004s PASS -ok github.com/gtsteffaniak/filebrowser/runner 0.003s +ok github.com/gtsteffaniak/filebrowser/backend/settings 0.005s +? github.com/gtsteffaniak/filebrowser/backend/share [no test files] +? github.com/gtsteffaniak/filebrowser/backend/storage [no test files] +? github.com/gtsteffaniak/filebrowser/backend/storage/bolt [no test files] +? github.com/gtsteffaniak/filebrowser/backend/swagger/docs [no test files] PASS -ok github.com/gtsteffaniak/filebrowser/settings 0.004s -? github.com/gtsteffaniak/filebrowser/share [no test files] -? github.com/gtsteffaniak/filebrowser/storage [no test files] -? github.com/gtsteffaniak/filebrowser/storage/bolt [no test files] +ok github.com/gtsteffaniak/filebrowser/backend/users 0.003s PASS -ok github.com/gtsteffaniak/filebrowser/users 0.002s -? github.com/gtsteffaniak/filebrowser/utils [no test files] -? github.com/gtsteffaniak/filebrowser/version [no test files] +ok github.com/gtsteffaniak/filebrowser/backend/utils 0.002s +? github.com/gtsteffaniak/filebrowser/backend/version [no test files] diff --git a/backend/cmd/root.go b/backend/cmd/root.go index 5132ebde..bca41196 100644 --- a/backend/cmd/root.go +++ b/backend/cmd/root.go @@ -31,13 +31,14 @@ func getStore(config string) (*storage.Storage, bool) { } func generalUsage() { - fmt.Printf(`usage: ./html-web-crawler [options] --urls + fmt.Printf(`usage: ./filebrowser [options] commands: - collect Collect data from URLs - crawl Crawl URLs and collect data - install Install chrome browser for javascript enabled scraping. - Note: Consider instead to install via native package manager, - then set "CHROME_EXECUTABLE" in the environment + -v Print the version + -c Print the default config file + set -u Username and password for the new user + set -a Create user as admin + set -s Specify a user scope + set -h Print this help message ` + "\n") } @@ -122,14 +123,24 @@ func StartFilebrowser() { log.Printf("Using Config file : %v", configPath) log.Println("Embeded frontend :", os.Getenv("FILEBROWSER_NO_EMBEDED") != "true") log.Println(database) - log.Println("Sources :", settings.Config.Server.Root) + sources := []string{} + for _, v := range settings.Config.Server.Sources { + sources = append(sources, v.Name+": "+v.Path) + } + log.Println("Sources :", sources) serverConfig := settings.Config.Server swagInfo := docs.SwaggerInfo swagInfo.BasePath = serverConfig.BaseURL swag.Register(docs.SwaggerInfo.InstanceName(), swagInfo) // initialize indexing and schedule indexing ever n minutes (default 5) - go files.InitializeIndex(serverConfig.Indexing) + sourceConfigs := settings.Config.Server.Sources + if len(sourceConfigs) == 0 { + log.Fatal("No sources configured, exiting...") + } + for _, source := range sourceConfigs { + go files.Initialize(source) + } if err := rootCMD(store, &serverConfig); err != nil { log.Fatal("Error starting filebrowser:", err) } diff --git a/backend/config.yaml b/backend/config.yaml index 6fea8512..c3d25272 100644 --- a/backend/config.yaml +++ b/backend/config.yaml @@ -1,7 +1,6 @@ server: port: 80 baseURL: "/" - root: "/srv" auth: method: password signup: false diff --git a/backend/files/file.go b/backend/files/file.go index a3cba8ab..b3bb6801 100644 --- a/backend/files/file.go +++ b/backend/files/file.go @@ -23,9 +23,7 @@ import ( "github.com/gtsteffaniak/filebrowser/backend/errors" "github.com/gtsteffaniak/filebrowser/backend/fileutils" - "github.com/gtsteffaniak/filebrowser/backend/settings" "github.com/gtsteffaniak/filebrowser/backend/users" - "github.com/gtsteffaniak/filebrowser/backend/utils" ) var ( @@ -57,11 +55,13 @@ type ExtendedFileInfo struct { Subtitles []string `json:"subtitles,omitempty"` Checksums map[string]string `json:"checksums,omitempty"` Token string `json:"token,omitempty"` + RealPath string `json:"-"` } // FileOptions are the options when getting a file info. type FileOptions struct { Path string // realpath + Source string IsDir bool Modify bool Expand bool @@ -75,9 +75,15 @@ func (f FileOptions) Components() (string, string) { } func FileInfoFaster(opts FileOptions) (ExtendedFileInfo, error) { - index := GetIndex(rootPath) - opts.Path = index.makeIndexPath(opts.Path) response := ExtendedFileInfo{} + if opts.Source == "" { + opts.Source = "default" + } + index := GetIndex(opts.Source) + if index == nil { + return response, fmt.Errorf("could not get index: %v ", opts.Source) + } + opts.Path = index.makeIndexPath(opts.Path) // Lock access for the specific path pathMutex := getMutex(opts.Path) pathMutex.Lock() @@ -86,7 +92,7 @@ func FileInfoFaster(opts FileOptions) (ExtendedFileInfo, error) { return response, os.ErrPermission } - _, isDir, err := GetRealPath(opts.Path) + realPath, isDir, err := index.GetRealPath(opts.Path) if err != nil { return response, err } @@ -120,13 +126,14 @@ func FileInfoFaster(opts FileOptions) (ExtendedFileInfo, error) { return response, err } if opts.Content { - content, err := getContent(opts.Path) + content, err := getContent("default", opts.Path) if err != nil { return response, err } response.Content = content } response.FileInfo = info + response.RealPath = realPath return response, nil } @@ -160,49 +167,13 @@ func GetChecksum(fullPath, algo string) (map[string]string, error) { return subs, nil } -// RealPath gets the real path for the file, resolving symlinks if supported. -func (i *FileInfo) RealPath() string { - realPath, _, _ := GetRealPath(rootPath, i.Path) - realPath, err := filepath.EvalSymlinks(realPath) - if err == nil { - return realPath - } - return i.Path -} - -func GetRealPath(relativePath ...string) (string, bool, error) { - combined := []string{settings.Config.Server.Root} - for _, path := range relativePath { - combined = append(combined, strings.TrimPrefix(path, settings.Config.Server.Root)) - } - joinedPath := filepath.Join(combined...) - - isDir, _ := utils.RealPathCache.Get(joinedPath + ":isdir").(bool) - cached, ok := utils.RealPathCache.Get(joinedPath).(string) - if ok && cached != "" { - return cached, isDir, nil - } - // Convert relative path to absolute path - absolutePath, err := filepath.Abs(joinedPath) - if err != nil { - return absolutePath, false, fmt.Errorf("could not get real path: %v, %s", combined, err) - } - // Resolve symlinks and get the real path - realPath, isDir, err := resolveSymlinks(absolutePath) - if err == nil { - utils.RealPathCache.Set(joinedPath, realPath) - utils.RealPathCache.Set(joinedPath+":isdir", isDir) - } - return realPath, isDir, err -} - -func DeleteFiles(absPath string, opts FileOptions) error { +func DeleteFiles(source, absPath string, dirPath string) error { err := os.RemoveAll(absPath) if err != nil { return err } - index := GetIndex(rootPath) - refreshConfig := FileOptions{Path: filepath.Dir(opts.Path), IsDir: true} + index := GetIndex(source) + refreshConfig := FileOptions{Path: dirPath, IsDir: true} err = index.RefreshFileInfo(refreshConfig) if err != nil { return err @@ -210,12 +181,12 @@ func DeleteFiles(absPath string, opts FileOptions) error { return nil } -func MoveResource(realsrc, realdst string, isSrcDir bool) error { +func MoveResource(source, realsrc, realdst string, isSrcDir bool) error { err := fileutils.MoveFile(realsrc, realdst) if err != nil { return err } - index := GetIndex(rootPath) + index := GetIndex(source) // refresh info for source and dest err = index.RefreshFileInfo(FileOptions{ Path: realsrc, @@ -235,12 +206,12 @@ func MoveResource(realsrc, realdst string, isSrcDir bool) error { return nil } -func CopyResource(realsrc, realdst string, isSrcDir bool) error { +func CopyResource(source, realsrc, realdst string, isSrcDir bool) error { err := fileutils.CopyFile(realsrc, realdst) if err != nil { return err } - index := GetIndex(rootPath) + index := GetIndex(source) refreshConfig := FileOptions{Path: realdst, IsDir: true} if !isSrcDir { refreshConfig.Path = filepath.Dir(realdst) @@ -253,14 +224,14 @@ func CopyResource(realsrc, realdst string, isSrcDir bool) error { } func WriteDirectory(opts FileOptions) error { - realPath, _, _ := GetRealPath(rootPath, opts.Path) + idx := GetIndex(opts.Source) + realPath, _, _ := idx.GetRealPath(opts.Path) // Ensure the parent directories exist err := os.MkdirAll(realPath, 0775) if err != nil { return err } - index := GetIndex(rootPath) - err = index.RefreshFileInfo(opts) + err = idx.RefreshFileInfo(opts) if err != nil { return errors.ErrEmptyKey } @@ -268,7 +239,8 @@ func WriteDirectory(opts FileOptions) error { } func WriteFile(opts FileOptions, in io.Reader) error { - dst, _, _ := GetRealPath(rootPath, opts.Path) + idx := GetIndex(opts.Source) + dst, _, _ := idx.GetRealPath(opts.Path) parentDir := filepath.Dir(dst) // Create the directory and all necessary parents err := os.MkdirAll(parentDir, 0775) @@ -290,8 +262,7 @@ func WriteFile(opts FileOptions, in io.Reader) error { } opts.Path = parentDir opts.IsDir = true - index := GetIndex(rootPath) - return index.RefreshFileInfo(opts) + return idx.RefreshFileInfo(opts) } // resolveSymlinks resolves symlinks in the given path @@ -323,8 +294,9 @@ func resolveSymlinks(path string) (string, bool, error) { } // addContent reads and sets content based on the file type. -func getContent(path string) (string, error) { - realPath, _, err := GetRealPath(rootPath, path) +func getContent(source, path string) (string, error) { + idx := GetIndex(source) + realPath, _, err := idx.GetRealPath(path) if err != nil { return "", err } @@ -344,7 +316,7 @@ func getContent(path string) (string, error) { } // DetectType detects the MIME type of a file and updates the ItemInfo struct. -func (i *ItemInfo) DetectType(path string, saveContent bool) { +func (i *ItemInfo) DetectType(realPath string, saveContent bool) { name := i.Name ext := filepath.Ext(name) @@ -354,9 +326,8 @@ func (i *ItemInfo) DetectType(path string, saveContent bool) { i.Type = extendedMimeTypeCheck(ext) } if i.Type == "blob" { - realpath, _, _ := GetRealPath(path) // Read only the first 512 bytes for efficient MIME detection - file, err := os.Open(realpath) + file, err := os.Open(realPath) if err != nil { } else { @@ -441,9 +412,11 @@ func Exists(path string) bool { func (info *FileInfo) SortItems() { sort.Slice(info.Folders, func(i, j int) bool { + nameWithoutExt := strings.Split(info.Folders[i].Name, ".")[0] + nameWithoutExt2 := strings.Split(info.Folders[j].Name, ".")[0] // Convert strings to integers for numeric sorting if both are numeric - numI, errI := strconv.Atoi(info.Folders[i].Name) - numJ, errJ := strconv.Atoi(info.Folders[j].Name) + numI, errI := strconv.Atoi(nameWithoutExt) + numJ, errJ := strconv.Atoi(nameWithoutExt2) if errI == nil && errJ == nil { return numI < numJ } @@ -451,9 +424,11 @@ func (info *FileInfo) SortItems() { return strings.ToLower(info.Folders[i].Name) < strings.ToLower(info.Folders[j].Name) }) sort.Slice(info.Files, func(i, j int) bool { + nameWithoutExt := strings.Split(info.Files[i].Name, ".")[0] + nameWithoutExt2 := strings.Split(info.Files[j].Name, ".")[0] // Convert strings to integers for numeric sorting if both are numeric - numI, errI := strconv.Atoi(info.Files[i].Name) - numJ, errJ := strconv.Atoi(info.Files[j].Name) + numI, errI := strconv.Atoi(nameWithoutExt) + numJ, errJ := strconv.Atoi(nameWithoutExt2) if errI == nil && errJ == nil { return numI < numJ } diff --git a/backend/files/file_test.go b/backend/files/file_test.go index bd4e71e0..0e509a3a 100644 --- a/backend/files/file_test.go +++ b/backend/files/file_test.go @@ -1,11 +1,14 @@ package files import ( + "fmt" "os" "path/filepath" "reflect" "strings" "testing" + + "github.com/gtsteffaniak/filebrowser/backend/settings" ) func Test_GetRealPath(t *testing.T) { @@ -13,7 +16,7 @@ func Test_GetRealPath(t *testing.T) { if err != nil { return } - trimPrefix := filepath.Dir(filepath.Dir(cwd)) + "/" + trimPrefix := filepath.Dir(filepath.Dir(cwd)) tests := []struct { name string paths []string @@ -31,20 +34,20 @@ func Test_GetRealPath(t *testing.T) { path string isDir bool }{ - path: "backend/files", + path: "", isDir: true, }, }, { name: "current directory", paths: []string{ - "./file.go", + "./files/file.go", }, want: struct { path string isDir bool }{ - path: "backend/files/file.go", + path: "/files/file.go", isDir: false, }, }, @@ -62,9 +65,16 @@ func Test_GetRealPath(t *testing.T) { }, }, } + idx := Index{ + Source: settings.Source{ + Path: trimPrefix, + }, + } + for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - realPath, isDir, _ := GetRealPath(tt.paths...) + realPath, isDir, _ := idx.GetRealPath(tt.paths...) + fmt.Println(realPath, trimPrefix) adjustedRealPath := strings.TrimPrefix(realPath, trimPrefix) if tt.want.path != adjustedRealPath || tt.want.isDir != isDir { t.Errorf("expected %v:%v but got: %v:%v", tt.want.path, tt.want.isDir, adjustedRealPath, isDir) @@ -83,30 +93,30 @@ func TestSortItems(t *testing.T) { name: "Numeric and Lexicographical Sorting", input: FileInfo{ Folders: []ItemInfo{ - {Name: "10"}, - {Name: "2"}, + {Name: "10.txt"}, + {Name: "2.txt"}, {Name: "apple"}, {Name: "Banana"}, }, Files: []ItemInfo{ - {Name: "File2"}, - {Name: "File10"}, + {Name: "File2.txt"}, + {Name: "File10.txt"}, {Name: "File1"}, {Name: "banana"}, }, }, expected: FileInfo{ Folders: []ItemInfo{ - {Name: "2"}, - {Name: "10"}, + {Name: "2.txt"}, + {Name: "10.txt"}, {Name: "apple"}, {Name: "Banana"}, }, Files: []ItemInfo{ {Name: "banana"}, {Name: "File1"}, - {Name: "File10"}, - {Name: "File2"}, + {Name: "File10.txt"}, + {Name: "File2.txt"}, }, }, }, @@ -114,8 +124,8 @@ func TestSortItems(t *testing.T) { name: "Only Lexicographical Sorting", input: FileInfo{ Folders: []ItemInfo{ - {Name: "dog"}, - {Name: "Cat"}, + {Name: "dog.txt"}, + {Name: "Cat.txt"}, {Name: "apple"}, }, Files: []ItemInfo{ @@ -127,8 +137,8 @@ func TestSortItems(t *testing.T) { expected: FileInfo{ Folders: []ItemInfo{ {Name: "apple"}, - {Name: "Cat"}, - {Name: "dog"}, + {Name: "Cat.txt"}, + {Name: "dog.txt"}, }, Files: []ItemInfo{ {Name: "apple"}, diff --git a/backend/files/indexingFiles.go b/backend/files/indexingFiles.go index 5a47eace..125fc143 100644 --- a/backend/files/indexingFiles.go +++ b/backend/files/indexingFiles.go @@ -5,6 +5,7 @@ import ( "log" "os" "path/filepath" + "slices" "strings" "sync" "time" @@ -14,7 +15,7 @@ import ( ) type Index struct { - Root string + settings.Source Directories map[string]*FileInfo NumDirs uint64 NumFiles uint64 @@ -30,32 +31,42 @@ type Index struct { } var ( - rootPath string = "/srv" - indexes []*Index + indexes map[string]*Index indexesMutex sync.RWMutex + RootPaths map[string]string ) -func InitializeIndex(enabled bool) { - if enabled { +func Initialize(source settings.Source) { + indexesMutex.RLock() + newIndex := Index{ + Source: source, + Directories: make(map[string]*FileInfo), + } + if RootPaths == nil { + RootPaths = make(map[string]string) + } + RootPaths[source.Name] = source.Path + indexes = make(map[string]*Index) + indexes[newIndex.Source.Name] = &newIndex + indexesMutex.RUnlock() + + if !newIndex.Source.Config.Disabled { time.Sleep(time.Second) - if settings.Config.Server.Root != "" { - rootPath = settings.Config.Server.Root - } - si := GetIndex(rootPath) log.Println("Initializing index and assessing file system complexity") - si.RunIndexing("/", false) - go si.setupIndexingScanners() + newIndex.RunIndexing("/", false) + go newIndex.setupIndexingScanners() + } else { + log.Println("Indexing disabled for source: ", newIndex.Source.Name) } } // Define a function to recursively index files and directories -func (si *Index) indexDirectory(adjustedPath string, quick, recursive bool) error { - realPath := strings.TrimRight(si.Root, "/") + adjustedPath - +func (idx *Index) indexDirectory(adjustedPath string, quick, recursive bool) error { + realPath := strings.TrimRight(idx.Source.Path, "/") + adjustedPath // Open the directory dir, err := os.Open(realPath) if err != nil { - si.RemoveDirectory(adjustedPath) // Remove, must have been deleted + idx.RemoveDirectory(adjustedPath) // Remove, must have been deleted return err } defer dir.Close() @@ -69,21 +80,21 @@ func (si *Index) indexDirectory(adjustedPath string, quick, recursive bool) erro combinedPath = "/" } // get whats currently in cache - si.mu.RLock() + idx.mu.RLock() cacheDirItems := []ItemInfo{} modChange := true // default to true - cachedDir, exists := si.Directories[adjustedPath] + cachedDir, exists := idx.Directories[adjustedPath] if exists && quick { modChange = dirInfo.ModTime() != cachedDir.ModTime cacheDirItems = cachedDir.Folders } - si.mu.RUnlock() + idx.mu.RUnlock() // If the directory has not been modified since the last index, skip expensive readdir // recursively check cached dirs for mod time changes as well if !modChange && recursive { for _, item := range cacheDirItems { - err = si.indexDirectory(combinedPath+item.Name, quick, true) + err = idx.indexDirectory(combinedPath+item.Name, quick, true) if err != nil { fmt.Printf("error indexing directory %v : %v", combinedPath+item.Name, err) } @@ -92,9 +103,9 @@ func (si *Index) indexDirectory(adjustedPath string, quick, recursive bool) erro } if quick { - si.mu.Lock() - si.FilesChangedDuringIndexing = true - si.mu.Unlock() + idx.mu.Lock() + idx.FilesChangedDuringIndexing = true + idx.mu.Unlock() } // Read directory contents @@ -109,36 +120,56 @@ func (si *Index) indexDirectory(adjustedPath string, quick, recursive bool) erro // Process each file and directory in the current directory for _, file := range files { + + isDir := file.IsDir() + fullCombined := combinedPath + file.Name() + if idx.shouldSkip(isDir, isHidden(file, ""), fullCombined) { + continue + } itemInfo := &ItemInfo{ Name: file.Name(), ModTime: file.ModTime(), } - if file.IsDir() { + + // fix for .app files on macos which are technically directories, but we don't want to treat them as such + if isDir && strings.HasSuffix(file.Name(), ".app") { + isDir = false + } + if isDir { + + // skip non-indexable dirs. + if file.Name() == "$RECYCLE.BIN" || file.Name() == "System Volume Information" { + continue + } + dirPath := combinedPath + file.Name() if recursive { // Recursively index the subdirectory - err = si.indexDirectory(dirPath, quick, recursive) + err = idx.indexDirectory(dirPath, quick, recursive) if err != nil { log.Printf("Failed to index directory %s: %v", dirPath, err) continue } } - realDirInfo, exists := si.GetMetadataInfo(dirPath, true) + realDirInfo, exists := idx.GetMetadataInfo(dirPath, true) if exists { itemInfo.Size = realDirInfo.Size } totalSize += itemInfo.Size itemInfo.Type = "directory" dirInfos = append(dirInfos, *itemInfo) - si.NumDirs++ + idx.NumDirs++ } else { - itemInfo.DetectType(combinedPath+file.Name(), false) + itemInfo.DetectType(fullCombined, false) itemInfo.Size = file.Size() fileInfos = append(fileInfos, *itemInfo) totalSize += itemInfo.Size - si.NumFiles++ + idx.NumFiles++ } } + if totalSize == 0 && idx.Source.Config.IgnoreZeroSizeFolders { + return nil + } // Create FileInfo for the current directory dirFileInfo := &FileInfo{ Path: adjustedPath, @@ -155,22 +186,23 @@ func (si *Index) indexDirectory(adjustedPath string, quick, recursive bool) erro dirFileInfo.SortItems() // Update the current directory metadata in the index - si.UpdateMetadata(dirFileInfo) + idx.UpdateMetadata(dirFileInfo) return nil } -func (si *Index) makeIndexPath(subPath string) string { +func (idx *Index) makeIndexPath(subPath string) string { + subPath = strings.ReplaceAll(subPath, "\\", "/") if strings.HasPrefix(subPath, "./") { subPath = strings.TrimPrefix(subPath, ".") } - if si.Root == subPath || subPath == "." { + if idx.Source.Path == subPath || subPath == "." { return "/" } // clean path subPath = strings.TrimSuffix(subPath, "/") // remove index prefix - adjustedPath := strings.TrimPrefix(subPath, si.Root) + adjustedPath := strings.TrimPrefix(subPath, idx.Source.Path) // remove trailing slash adjustedPath = strings.TrimSuffix(adjustedPath, "/") if !strings.HasPrefix(adjustedPath, "/") { @@ -179,43 +211,65 @@ func (si *Index) makeIndexPath(subPath string) string { return adjustedPath } -func (si *Index) recursiveUpdateDirSizes(childInfo *FileInfo, previousSize int64) { +func (idx *Index) recursiveUpdateDirSizes(childInfo *FileInfo, previousSize int64) { parentDir := utils.GetParentDirectoryPath(childInfo.Path) - parentInfo, exists := si.GetMetadataInfo(parentDir, true) + parentInfo, exists := idx.GetMetadataInfo(parentDir, true) if !exists || parentDir == "" { return } newSize := parentInfo.Size - previousSize + childInfo.Size parentInfo.Size += newSize - si.UpdateMetadata(parentInfo) - si.recursiveUpdateDirSizes(parentInfo, newSize) + idx.UpdateMetadata(parentInfo) + idx.recursiveUpdateDirSizes(parentInfo, newSize) } -func (si *Index) RefreshFileInfo(opts FileOptions) error { +func (idx *Index) GetRealPath(relativePath ...string) (string, bool, error) { + combined := append([]string{idx.Source.Path}, relativePath...) + joinedPath := filepath.Join(combined...) + isDir, _ := utils.RealPathCache.Get(joinedPath + ":isdir").(bool) + cached, ok := utils.RealPathCache.Get(joinedPath).(string) + if ok && cached != "" { + return cached, isDir, nil + } + // Convert relative path to absolute path + absolutePath, err := filepath.Abs(joinedPath) + if err != nil { + return absolutePath, false, fmt.Errorf("could not get real path: %v, %s", joinedPath, err) + } + // Resolve symlinks and get the real path + realPath, isDir, err := resolveSymlinks(absolutePath) + if err == nil { + utils.RealPathCache.Set(joinedPath, realPath) + utils.RealPathCache.Set(joinedPath+":isdir", isDir) + } + return realPath, isDir, err +} + +func (idx *Index) RefreshFileInfo(opts FileOptions) error { refreshOptions := FileOptions{ Path: opts.Path, IsDir: opts.IsDir, } if !refreshOptions.IsDir { - refreshOptions.Path = si.makeIndexPath(filepath.Dir(refreshOptions.Path)) + refreshOptions.Path = idx.makeIndexPath(filepath.Dir(refreshOptions.Path)) refreshOptions.IsDir = true } else { - refreshOptions.Path = si.makeIndexPath(refreshOptions.Path) + refreshOptions.Path = idx.makeIndexPath(refreshOptions.Path) } - err := si.indexDirectory(refreshOptions.Path, false, false) + err := idx.indexDirectory(refreshOptions.Path, false, false) if err != nil { return fmt.Errorf("file/folder does not exist to refresh data: %s", refreshOptions.Path) } - file, exists := si.GetMetadataInfo(refreshOptions.Path, true) + file, exists := idx.GetMetadataInfo(refreshOptions.Path, true) if !exists { return fmt.Errorf("file/folder does not exist in metadata: %s", refreshOptions.Path) } - current, firstExisted := si.GetMetadataInfo(refreshOptions.Path, true) + current, firstExisted := idx.GetMetadataInfo(refreshOptions.Path, true) refreshParentInfo := firstExisted && current.Size != file.Size //utils.PrintStructFields(*file) - result := si.UpdateMetadata(file) + result := idx.UpdateMetadata(file) if !result { return fmt.Errorf("file/folder does not exist in metadata: %s", refreshOptions.Path) } @@ -223,7 +277,37 @@ func (si *Index) RefreshFileInfo(opts FileOptions) error { return nil } if refreshParentInfo { - si.recursiveUpdateDirSizes(file, current.Size) + idx.recursiveUpdateDirSizes(file, current.Size) } return nil } + +func isHidden(file os.FileInfo, realpath string) bool { + return file.Name()[0] == '.' +} + +func (idx *Index) shouldSkip(isDir bool, isHidden bool, fullCombined string) bool { + // check inclusions first + if isDir && len(idx.Source.Config.Include.Folders) > 0 { + if !slices.Contains(idx.Source.Config.Include.Folders, fullCombined) { + return true + } + } + if !isDir && len(idx.Source.Config.Include.Files) > 0 { + if !slices.Contains(idx.Source.Config.Include.Files, fullCombined) { + return true + } + } + + // check exclusions + if isDir && slices.Contains(idx.Source.Config.Exclude.Folders, fullCombined) { + return true + } + if !isDir && slices.Contains(idx.Source.Config.Exclude.Files, fullCombined) { + return true + } + if idx.Source.Config.IgnoreHidden && isHidden { + return true + } + return false +} diff --git a/backend/files/indexingSchedule.go b/backend/files/indexingSchedule.go index 442e24de..cc58707f 100644 --- a/backend/files/indexingSchedule.go +++ b/backend/files/indexingSchedule.go @@ -3,8 +3,6 @@ package files import ( "log" "time" - - "github.com/gtsteffaniak/filebrowser/backend/settings" ) // schedule in minutes @@ -19,99 +17,99 @@ var scanSchedule = []time.Duration{ 4 * time.Hour, // 4 hours for quick scan & 20 hours for a full scan } -func (si *Index) newScanner(origin string) { +func (idx *Index) newScanner(origin string) { fullScanAnchor := 3 fullScanCounter := 0 // every 5th scan is a full scan for { // Determine sleep time with modifiers fullScanCounter++ - sleepTime := scanSchedule[si.currentSchedule] + si.SmartModifier - if si.assessment == "simple" { - sleepTime = scanSchedule[si.currentSchedule] - si.SmartModifier + sleepTime := scanSchedule[idx.currentSchedule] + idx.SmartModifier + if idx.assessment == "simple" { + sleepTime = scanSchedule[idx.currentSchedule] - idx.SmartModifier } - if settings.Config.Server.IndexingInterval > 0 { - sleepTime = time.Duration(settings.Config.Server.IndexingInterval) * time.Minute + if idx.Source.Config.IndexingInterval > 0 { + sleepTime = time.Duration(idx.Source.Config.IndexingInterval) * time.Minute } // Log and sleep before indexing log.Printf("Next scan in %v\n", sleepTime) time.Sleep(sleepTime) - si.scannerMu.Lock() + idx.scannerMu.Lock() if fullScanCounter == 5 { - si.RunIndexing(origin, false) // Full scan + idx.RunIndexing(origin, false) // Full scan fullScanCounter = 0 } else { - si.RunIndexing(origin, true) // Quick scan + idx.RunIndexing(origin, true) // Quick scan } - si.scannerMu.Unlock() + idx.scannerMu.Unlock() // Adjust schedule based on file changes - if si.FilesChangedDuringIndexing { + if idx.FilesChangedDuringIndexing { // Move to at least the full-scan anchor or reduce interval - if si.currentSchedule > fullScanAnchor { - si.currentSchedule = fullScanAnchor - } else if si.currentSchedule > 0 { - si.currentSchedule-- + if idx.currentSchedule > fullScanAnchor { + idx.currentSchedule = fullScanAnchor + } else if idx.currentSchedule > 0 { + idx.currentSchedule-- } } else { // Increment toward the longest interval if no changes - if si.currentSchedule < len(scanSchedule)-1 { - si.currentSchedule++ + if idx.currentSchedule < len(scanSchedule)-1 { + idx.currentSchedule++ } } - if si.assessment == "simple" && si.currentSchedule > 3 { - si.currentSchedule = 3 + if idx.assessment == "simple" && idx.currentSchedule > 3 { + idx.currentSchedule = 3 } // Ensure `currentSchedule` stays within bounds - if si.currentSchedule < 0 { - si.currentSchedule = 0 - } else if si.currentSchedule >= len(scanSchedule) { - si.currentSchedule = len(scanSchedule) - 1 + if idx.currentSchedule < 0 { + idx.currentSchedule = 0 + } else if idx.currentSchedule >= len(scanSchedule) { + idx.currentSchedule = len(scanSchedule) - 1 } } } -func (si *Index) RunIndexing(origin string, quick bool) { - prevNumDirs := si.NumDirs - prevNumFiles := si.NumFiles +func (idx *Index) RunIndexing(origin string, quick bool) { + prevNumDirs := idx.NumDirs + prevNumFiles := idx.NumFiles if quick { log.Println("Starting quick scan") } else { log.Println("Starting full scan") - si.NumDirs = 0 - si.NumFiles = 0 + idx.NumDirs = 0 + idx.NumFiles = 0 } startTime := time.Now() - si.FilesChangedDuringIndexing = false + idx.FilesChangedDuringIndexing = false // Perform the indexing operation - err := si.indexDirectory("/", quick, true) + err := idx.indexDirectory("/", quick, true) if err != nil { log.Printf("Error during indexing: %v", err) } // Update the LastIndexed time - si.LastIndexed = time.Now() - si.indexingTime = int(time.Since(startTime).Seconds()) + idx.LastIndexed = time.Now() + idx.indexingTime = int(time.Since(startTime).Seconds()) if !quick { // update smart indexing - if si.indexingTime < 3 || si.NumDirs < 10000 { - si.assessment = "simple" - si.SmartModifier = 4 * time.Minute - } else if si.indexingTime > 120 || si.NumDirs > 500000 { - si.assessment = "complex" - modifier := si.indexingTime / 10 // seconds - si.SmartModifier = time.Duration(modifier) * time.Minute + if idx.indexingTime < 3 || idx.NumDirs < 10000 { + idx.assessment = "simple" + idx.SmartModifier = 4 * time.Minute + } else if idx.indexingTime > 120 || idx.NumDirs > 500000 { + idx.assessment = "complex" + modifier := idx.indexingTime / 10 // seconds + idx.SmartModifier = time.Duration(modifier) * time.Minute } else { - si.assessment = "normal" + idx.assessment = "normal" } - log.Printf("Index assessment : complexity=%v directories=%v files=%v \n", si.assessment, si.NumDirs, si.NumFiles) - if si.NumDirs != prevNumDirs || si.NumFiles != prevNumFiles { - si.FilesChangedDuringIndexing = true + log.Printf("Index assessment : complexity=%v directories=%v files=%v \n", idx.assessment, idx.NumDirs, idx.NumFiles) + if idx.NumDirs != prevNumDirs || idx.NumFiles != prevNumFiles { + idx.FilesChangedDuringIndexing = true } } - log.Printf("Time Spent Indexing : %v seconds\n", si.indexingTime) + log.Printf("Time Spent Indexing : %v seconds\n", idx.indexingTime) } -func (si *Index) setupIndexingScanners() { - go si.newScanner("/") +func (idx *Index) setupIndexingScanners() { + go idx.newScanner("/") } diff --git a/backend/files/indexing_test.go b/backend/files/indexing_test.go index 6e71df97..d7b74916 100644 --- a/backend/files/indexing_test.go +++ b/backend/files/indexing_test.go @@ -11,16 +11,19 @@ import ( ) func BenchmarkFillIndex(b *testing.B) { - InitializeIndex(false) - si := GetIndex(settings.Config.Server.Root) + Initialize(settings.Source{ + Name: "test", + Path: "/srv", + }) + idx := GetIndex("test") b.ResetTimer() b.ReportAllocs() for i := 0; i < b.N; i++ { - si.createMockData(50, 3) // 1000 dirs, 3 files per dir + idx.createMockData(50, 3) // 1000 dirs, 3 files per dir } } -func (si *Index) createMockData(numDirs, numFilesPerDir int) { +func (idx *Index) createMockData(numDirs, numFilesPerDir int) { for i := 0; i < numDirs; i++ { dirPath := generateRandomPath(rand.Intn(3) + 1) files := []ItemInfo{} // Slice of FileInfo @@ -40,7 +43,7 @@ func (si *Index) createMockData(numDirs, numFilesPerDir int) { Files: files, } - si.UpdateMetadata(dirInfo) + idx.UpdateMetadata(dirInfo) } } @@ -78,6 +81,7 @@ func TestMakeIndexPath(t *testing.T) { subPath string expected string }{ + // Linux {"Root path returns slash", "/", "/"}, {"Dot-prefixed returns slash", ".", "/"}, {"Double-dot prefix ignored", "./", "/"}, @@ -87,15 +91,114 @@ func TestMakeIndexPath(t *testing.T) { {"Trailing slash removed", "/test/", "/test"}, {"Subpath without root prefix", "/other/test", "/other/test"}, {"Complex nested paths", "/nested/path", "/nested/path"}, + // Windows + {"Mixed slash", "/first\\second", "/first/second"}, + {"Windows slash", "\\first\\second", "/first/second"}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - si := &Index{Root: "/"} - result := si.makeIndexPath(tt.subPath) + idx := &Index{Source: settings.Source{Path: "/"}} + result := idx.makeIndexPath(tt.subPath) if result != tt.expected { - t.Errorf("makeIndexPath(%q) = %q; want %q", tt.name, result, tt.expected) + t.Errorf("makeIndexPath(%q)\ngot %q\nwant %q", tt.name, result, tt.expected) } }) } } + +func TestMakeIndexPathRoot(t *testing.T) { + tests := []struct { + name string + subPath string + expected string + }{ + // Linux + {"Root path returns slash", "/rootpath", "/"}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + idx := &Index{Source: settings.Source{Path: "/rootpath", Name: "default"}} + result := idx.makeIndexPath(tt.subPath) + if result != tt.expected { + t.Errorf("makeIndexPath(%q)\ngot %q\nwant %q", tt.name, result, tt.expected) + } + }) + } +} + +func BenchmarkCheckIndexExclude(b *testing.B) { + tests := []struct { + isDir bool + isHidden bool + fullPath string + }{ + {false, false, "/test/.test"}, + {true, false, "/test/.test"}, + {true, true, "/test/.test"}, + {false, false, "/test/filepath"}, + {false, true, "/test/filepath"}, + {true, true, "/test/filepath"}, + } + + b.ResetTimer() + b.ReportAllocs() + idx := Index{ + Source: settings.Source{ + Name: "files", + Config: settings.IndexConfig{ + IgnoreHidden: true, + Exclude: settings.IndexFilter{ + Files: []string{"test", "filepath", ".test", ".filepath", "test", "filepath", ".test", ".filepath"}, + Folders: []string{"test", "filepath", ".test", ".filepath", "test", "filepath", ".test", ".filepath"}, + FileEndsWith: []string{".zip", ".tar", ".jpeg"}, + }, + }, + }, + } + + for i := 0; i < b.N; i++ { + for _, v := range tests { + idx.shouldSkip(v.isDir, v.isHidden, v.fullPath) + } + } + +} +func BenchmarkCheckIndexConditionsInclude(b *testing.B) { + tests := []struct { + isDir bool + isHidden bool + fullPath string + }{ + {false, false, "/test/.test"}, + {true, false, "/test/.test"}, + {true, true, "/test/.test"}, + {false, false, "/test/filepath"}, + {false, true, "/test/filepath"}, + {true, true, "/test/filepath"}, + } + + b.ResetTimer() + b.ReportAllocs() + idx2 := Index{ + Source: settings.Source{ + Name: "files", + Config: settings.IndexConfig{ + IgnoreHidden: true, + Include: settings.IndexFilter{ + Files: []string{"test", "filepath", ".test", ".filepath", "test", "filepath", ".test", ".filepath"}, + Folders: []string{"test", "filepath", ".test", ".filepath", "test", "filepath", ".test", ".filepath"}, + FileEndsWith: []string{".zip", ".tar", ".jpeg"}, + }, + }, + }, + } + + for i := 0; i < b.N; i++ { + for _, v := range tests { + idx2.shouldSkip(v.isDir, v.isHidden, v.fullPath) + } + } + +} diff --git a/backend/files/search.go b/backend/files/search.go index 4f5986cc..a4a3ff14 100644 --- a/backend/files/search.go +++ b/backend/files/search.go @@ -20,21 +20,21 @@ type SearchResult struct { Size int64 `json:"size"` } -func (si *Index) Search(search string, scope string, sourceSession string) []SearchResult { +func (idx *Index) Search(search string, scope string, sourceSession string) []SearchResult { // Remove slashes - scope = si.makeIndexPath(scope) + scope = idx.makeIndexPath(scope) runningHash := utils.GenerateRandomHash(4) sessionInProgress.Store(sourceSession, runningHash) // Store the value in the sync.Map searchOptions := ParseSearch(search) results := make(map[string]SearchResult, 0) count := 0 var directories []string - cachedDirs, ok := utils.SearchResultsCache.Get(si.Root + scope).([]string) + cachedDirs, ok := utils.SearchResultsCache.Get(idx.Source.Path + scope).([]string) if ok { directories = cachedDirs } else { - directories = si.getDirsInScope(scope) - utils.SearchResultsCache.Set(si.Root+scope, directories) + directories = idx.getDirsInScope(scope) + utils.SearchResultsCache.Set(idx.Source.Path+scope, directories) } for _, searchTerm := range searchOptions.Terms { if searchTerm == "" { @@ -43,12 +43,12 @@ func (si *Index) Search(search string, scope string, sourceSession string) []Sea if count > maxSearchResults { break } - si.mu.Lock() + idx.mu.Lock() for _, dirName := range directories { scopedPath := strings.TrimPrefix(strings.TrimPrefix(dirName, scope), "/") + "/" - si.mu.Unlock() - dir, found := si.GetReducedMetadata(dirName, true) - si.mu.Lock() + idx.mu.Unlock() + dir, found := idx.GetReducedMetadata(dirName, true) + idx.mu.Lock() if !found { continue } @@ -74,7 +74,7 @@ func (si *Index) Search(search string, scope string, sourceSession string) []Sea } value, found := sessionInProgress.Load(sourceSession) if !found || value != runningHash { - si.mu.Unlock() + idx.mu.Unlock() return []SearchResult{} } if count > maxSearchResults { @@ -87,7 +87,7 @@ func (si *Index) Search(search string, scope string, sourceSession string) []Sea } } } - si.mu.Unlock() + idx.mu.Unlock() } // Sort keys based on the number of elements in the path after splitting by "/" @@ -169,11 +169,11 @@ func (fi ItemInfo) containsSearchTerm(searchTerm string, options SearchOptions) return true } -func (si *Index) getDirsInScope(scope string) []string { +func (idx *Index) getDirsInScope(scope string) []string { newList := []string{} - si.mu.Lock() - defer si.mu.Unlock() - for k := range si.Directories { + idx.mu.Lock() + defer idx.mu.Unlock() + for k := range idx.Directories { if strings.HasPrefix(k, scope) || scope == "" { newList = append(newList, k) } diff --git a/backend/files/search_test.go b/backend/files/search_test.go index 968b56ab..755335fc 100644 --- a/backend/files/search_test.go +++ b/backend/files/search_test.go @@ -4,14 +4,15 @@ import ( "reflect" "testing" + "github.com/gtsteffaniak/filebrowser/backend/settings" "github.com/stretchr/testify/assert" ) func BenchmarkSearchAllIndexes(b *testing.B) { - InitializeIndex(false) - si := GetIndex(rootPath) + Initialize(settings.Source{Name: "test", Path: "/srv"}) + idx := GetIndex("test") - si.createMockData(50, 3) // 50 dirs, 3 files per dir + idx.createMockData(50, 3) // 50 dirs, 3 files per dir // Generate 100 random search terms searchTerms := generateRandomSearchTerms(100) @@ -21,7 +22,7 @@ func BenchmarkSearchAllIndexes(b *testing.B) { for i := 0; i < b.N; i++ { // Execute the SearchAllIndexes function for _, term := range searchTerms { - si.Search(term, "/", "test") + idx.Search(term, "/", "test") } } } @@ -74,14 +75,14 @@ func TestParseSearch(t *testing.T) { } func TestSearchWhileIndexing(t *testing.T) { - InitializeIndex(false) - si := GetIndex(rootPath) + Initialize(settings.Source{Name: "test", Path: "/srv"}) + idx := GetIndex("test") searchTerms := generateRandomSearchTerms(10) for i := 0; i < 5; i++ { - go si.createMockData(100, 100) // Creating mock data concurrently + go idx.createMockData(100, 100) // Creating mock data concurrently for _, term := range searchTerms { - go si.Search(term, "/", "test") // Search concurrently + go idx.Search(term, "/", "test") // Search concurrently } } } diff --git a/backend/files/sync.go b/backend/files/sync.go index 5619826a..ab75d6c8 100644 --- a/backend/files/sync.go +++ b/backend/files/sync.go @@ -2,27 +2,25 @@ package files import ( "path/filepath" - - "github.com/gtsteffaniak/filebrowser/backend/settings" ) // UpdateFileMetadata updates the FileInfo for the specified directory in the index. -func (si *Index) UpdateMetadata(info *FileInfo) bool { - si.mu.Lock() - defer si.mu.Unlock() - si.Directories[info.Path] = info +func (idx *Index) UpdateMetadata(info *FileInfo) bool { + idx.mu.Lock() + defer idx.mu.Unlock() + idx.Directories[info.Path] = info return true } // GetMetadataInfo retrieves the FileInfo from the specified directory in the index. -func (si *Index) GetReducedMetadata(target string, isDir bool) (*FileInfo, bool) { - si.mu.Lock() - defer si.mu.Unlock() - checkDir := si.makeIndexPath(target) +func (idx *Index) GetReducedMetadata(target string, isDir bool) (*FileInfo, bool) { + idx.mu.Lock() + defer idx.mu.Unlock() + checkDir := idx.makeIndexPath(target) if !isDir { - checkDir = si.makeIndexPath(filepath.Dir(target)) + checkDir = idx.makeIndexPath(filepath.Dir(target)) } - dir, exists := si.Directories[checkDir] + dir, exists := idx.Directories[checkDir] if !exists { return nil, false } @@ -48,42 +46,30 @@ func (si *Index) GetReducedMetadata(target string, isDir bool) (*FileInfo, bool) } // GetMetadataInfo retrieves the FileInfo from the specified directory in the index. -func (si *Index) GetMetadataInfo(target string, isDir bool) (*FileInfo, bool) { - si.mu.RLock() - defer si.mu.RUnlock() - checkDir := si.makeIndexPath(target) +func (idx *Index) GetMetadataInfo(target string, isDir bool) (*FileInfo, bool) { + idx.mu.RLock() + defer idx.mu.RUnlock() + checkDir := idx.makeIndexPath(target) if !isDir { - checkDir = si.makeIndexPath(filepath.Dir(target)) + checkDir = idx.makeIndexPath(filepath.Dir(target)) } - dir, exists := si.Directories[checkDir] + dir, exists := idx.Directories[checkDir] return dir, exists } -func (si *Index) RemoveDirectory(path string) { - si.mu.Lock() - defer si.mu.Unlock() - si.NumDeleted++ - delete(si.Directories, path) +func (idx *Index) RemoveDirectory(path string) { + idx.mu.Lock() + defer idx.mu.Unlock() + idx.NumDeleted++ + delete(idx.Directories, path) } -func GetIndex(root string) *Index { - for _, index := range indexes { - if index.Root == root { - return index - } - } - if settings.Config.Server.Root != "" { - rootPath = settings.Config.Server.Root - } - newIndex := &Index{ - Root: rootPath, - Directories: map[string]*FileInfo{}, - NumDirs: 0, - NumFiles: 0, - } - newIndex.Directories["/"] = &FileInfo{} +func GetIndex(name string) *Index { indexesMutex.Lock() - indexes = append(indexes, newIndex) - indexesMutex.Unlock() - return newIndex + defer indexesMutex.Unlock() + index, ok := indexes[name] + if !ok { + return nil + } + return index } diff --git a/backend/files/sync_test.go b/backend/files/sync_test.go index 70a4ffad..f53d0725 100644 --- a/backend/files/sync_test.go +++ b/backend/files/sync_test.go @@ -3,6 +3,7 @@ package files import ( "testing" + "github.com/gtsteffaniak/filebrowser/backend/settings" "github.com/stretchr/testify/assert" ) @@ -207,7 +208,10 @@ func TestRemoveDirectory(t *testing.T) { func init() { testIndex = Index{ - Root: "/", + Source: settings.Source{ + Path: "/", + Name: "test", + }, NumFiles: 10, NumDirs: 5, Directories: map[string]*FileInfo{ diff --git a/backend/go.mod b/backend/go.mod index 57993455..6e933955 100644 --- a/backend/go.mod +++ b/backend/go.mod @@ -7,8 +7,7 @@ require ( github.com/disintegration/imaging v1.6.2 github.com/dsoprea/go-exif/v3 v3.0.1 github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568 - github.com/gabriel-vasile/mimetype v1.4.7 - github.com/goccy/go-yaml v1.15.7 + github.com/goccy/go-yaml v1.15.13 github.com/golang-jwt/jwt/v4 v4.5.1 github.com/google/go-cmp v0.6.0 github.com/shirou/gopsutil/v3 v3.24.5 @@ -18,7 +17,7 @@ require ( github.com/stretchr/testify v1.9.0 github.com/swaggo/http-swagger v1.3.4 github.com/swaggo/swag v1.16.4 - golang.org/x/crypto v0.30.0 + golang.org/x/crypto v0.31.0 golang.org/x/image v0.23.0 golang.org/x/text v0.21.0 ) @@ -38,13 +37,13 @@ require ( github.com/golang/snappy v0.0.4 // indirect github.com/inconshreveable/mousetrap v1.1.0 // indirect github.com/josharian/intern v1.0.0 // indirect - github.com/mailru/easyjson v0.7.7 // indirect + github.com/mailru/easyjson v0.9.0 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect github.com/swaggo/files v1.0.1 // indirect github.com/yusufpapurcu/wmi v1.2.4 // indirect go.etcd.io/bbolt v1.3.11 // indirect - golang.org/x/net v0.32.0 // indirect + golang.org/x/net v0.33.0 // indirect golang.org/x/sys v0.28.0 // indirect golang.org/x/tools v0.28.0 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect diff --git a/backend/go.sum b/backend/go.sum index 3175e34a..6855c652 100644 --- a/backend/go.sum +++ b/backend/go.sum @@ -30,8 +30,6 @@ github.com/dsoprea/go-utility/v2 v2.0.0-20221003172846-a3e1774ef349 h1:DilThiXje github.com/dsoprea/go-utility/v2 v2.0.0-20221003172846-a3e1774ef349/go.mod h1:4GC5sXji84i/p+irqghpPFZBF8tRN/Q7+700G0/DLe8= github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568 h1:BHsljHzVlRcyQhjrss6TZTdY2VfCqZPbv5k3iBFa2ZQ= github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI435gkrCt3MPfRiAkVrwSbHsst4LCFVfpJc= -github.com/gabriel-vasile/mimetype v1.4.7 h1:SKFKl7kD0RiPdbht0s7hFtjl489WcQ1VyPW8ZzUMYCA= -github.com/gabriel-vasile/mimetype v1.4.7/go.mod h1:GDlAgAyIRT27BhFl53XNAFtfjzOkLaF35JdEG0P7LtU= github.com/go-errors/errors v1.0.1/go.mod h1:f4zRHt4oKfwPJE5k8C9vpYG+aDHdBFUsgrm6/TyX73Q= github.com/go-errors/errors v1.0.2/go.mod h1:psDX2osz5VnTOnFWbDeWwS7yejl+uV3FEWEp4lssFEs= github.com/go-errors/errors v1.1.1/go.mod h1:psDX2osz5VnTOnFWbDeWwS7yejl+uV3FEWEp4lssFEs= @@ -49,8 +47,8 @@ github.com/go-openapi/spec v0.21.0 h1:LTVzPc3p/RzRnkQqLRndbAzjY0d0BCL72A6j3CdL9Z github.com/go-openapi/spec v0.21.0/go.mod h1:78u6VdPw81XU44qEWGhtr982gJ5BWg2c0I5XwVMotYk= github.com/go-openapi/swag v0.23.0 h1:vsEVJDUo2hPJ2tu0/Xc+4noaxyEffXNIs3cOULZ+GrE= github.com/go-openapi/swag v0.23.0/go.mod h1:esZ8ITTYEsH1V2trKHjAN8Ai7xHb8RV+YSZ577vPjgQ= -github.com/goccy/go-yaml v1.15.7 h1:L7XuKpd/A66X4w/dlk08lVfiIADdy79a1AzRoIefC98= -github.com/goccy/go-yaml v1.15.7/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA= +github.com/goccy/go-yaml v1.15.13 h1:Xd87Yddmr2rC1SLLTm2MNDcTjeO/GYo0JGiww6gSTDg= +github.com/goccy/go-yaml v1.15.13/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA= github.com/golang-jwt/jwt/v4 v4.5.1 h1:JdqV9zKUdtaa9gdPlywC3aeoEsR681PlKC+4F5gQgeo= github.com/golang-jwt/jwt/v4 v4.5.1/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0= github.com/golang/geo v0.0.0-20190916061304-5b978397cfec/go.mod h1:QZ0nwyI2jOfgRAoBvP+ab5aRr7c9x7lhGEJrKvBwjWI= @@ -80,8 +78,8 @@ github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= -github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0= -github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= +github.com/mailru/easyjson v0.9.0 h1:PrnmzHw7262yW8sTBwxi1PdJA3Iw/EKBa8psRf7d9a4= +github.com/mailru/easyjson v0.9.0/go.mod h1:1+xMtQp2MRNVL/V1bOzuP3aP8VNwRW55fQUto+XFtTU= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 h1:o4JXh1EVt9k/+g42oCprj/FisM4qX9L3sZB3upGN2ZU= @@ -116,8 +114,8 @@ go.etcd.io/bbolt v1.3.11 h1:yGEzV1wPz2yVCLsD8ZAiGHhHVlczyC9d1rP43/VCRJ0= go.etcd.io/bbolt v1.3.11/go.mod h1:dksAq7YMXoljX0xu6VF5DMZGbhYYoLUalEiSySYAS4I= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= -golang.org/x/crypto v0.30.0 h1:RwoQn3GkWiMkzlX562cLB7OxWvjH1L8xutO2WoJcRoY= -golang.org/x/crypto v0.30.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk= +golang.org/x/crypto v0.31.0 h1:ihbySMvVjLAeSH1IbfcRTkD/iNscyz8rGzjF/E5hV6U= +golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk= golang.org/x/image v0.0.0-20191009234506-e7c1f5e7dbb8/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0= golang.org/x/image v0.23.0 h1:HseQ7c2OpPKTPVzNjG5fwJsOTCiiwS4QdsYi5XU6H68= golang.org/x/image v0.23.0/go.mod h1:wJJBTdLfCCf3tiHa1fNxpZmUI4mmoZvwMCPP0ddoNKY= @@ -135,8 +133,8 @@ golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.0.0-20221002022538-bcab6841153b/go.mod h1:YDH+HFinaLZZlnHAfSS6ZXJJ9M9t4Dl22yv3iI2vPwk= golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= -golang.org/x/net v0.32.0 h1:ZqPmj8Kzc+Y6e0+skZsuACbx+wzMgo5MQsJh9Qd6aYI= -golang.org/x/net v0.32.0/go.mod h1:CwU0IoeOlnQQWJ6ioyFrfRuomB8GKF6KbYXZVyeXNfs= +golang.org/x/net v0.33.0 h1:74SYHlV8BIgHIFC/LrYkOGIwL19eTYXQ5wc6TBuO36I= +golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.10.0 h1:3NQrjDixjgGwUOCaF8w2+VYHv0Ve/vGYSbdkTa98gmQ= diff --git a/backend/http/auth.go b/backend/http/auth.go index cabf36c8..80eb2f2b 100644 --- a/backend/http/auth.go +++ b/backend/http/auth.go @@ -17,6 +17,7 @@ import ( "golang.org/x/crypto/bcrypt" "github.com/gtsteffaniak/filebrowser/backend/errors" + "github.com/gtsteffaniak/filebrowser/backend/files" "github.com/gtsteffaniak/filebrowser/backend/settings" "github.com/gtsteffaniak/filebrowser/backend/share" "github.com/gtsteffaniak/filebrowser/backend/users" @@ -126,7 +127,7 @@ func signupHandler(w http.ResponseWriter, r *http.Request) { user.Username = info.Username user.Password = info.Password - userHome, err := config.MakeUserDir(user.Username, user.Scope, config.Server.Root) + userHome, err := config.MakeUserDir(user.Username, user.Scope, files.RootPaths["default"]) if err != nil { log.Printf("create user: failed to mkdir user home dir: [%s]", userHome) http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError) diff --git a/backend/http/middleware.go b/backend/http/middleware.go index b2764e53..f6e18c30 100644 --- a/backend/http/middleware.go +++ b/backend/http/middleware.go @@ -97,7 +97,7 @@ func withUserHelper(fn handleFunc) handleFunc { if settings.Config.Auth.Method == "noauth" { var err error // Retrieve the user from the store and store it in the context - data.user, err = store.Users.Get(config.Server.Root, "admin") + data.user, err = store.Users.Get(files.RootPaths["default"], "admin") if err != nil { return http.StatusInternalServerError, err } @@ -127,7 +127,7 @@ func withUserHelper(fn handleFunc) handleFunc { w.Header().Add("X-Renew-Token", "true") } // Retrieve the user from the store and store it in the context - data.user, err = store.Users.Get(config.Server.Root, tk.BelongsTo) + data.user, err = store.Users.Get(files.RootPaths["default"], tk.BelongsTo) if err != nil { return http.StatusInternalServerError, err } diff --git a/backend/http/preview.go b/backend/http/preview.go index 08e24423..5a5b5258 100644 --- a/backend/http/preview.go +++ b/backend/http/preview.go @@ -9,6 +9,7 @@ import ( "os" "path/filepath" "strings" + "time" "github.com/gtsteffaniak/filebrowser/backend/files" "github.com/gtsteffaniak/filebrowser/backend/img" @@ -42,17 +43,18 @@ type FileCache interface { // @Router /api/preview [get] func previewHandler(w http.ResponseWriter, r *http.Request, d *requestContext) (int, error) { path := r.URL.Query().Get("path") + source := r.URL.Query().Get("source") previewSize := r.URL.Query().Get("size") if previewSize != "small" { previewSize = "large" } - if path == "" { return http.StatusBadRequest, fmt.Errorf("invalid request path") } response, err := files.FileInfoFaster(files.FileOptions{ Path: filepath.Join(d.user.Scope, path), Modify: d.user.Perm.Modify, + Source: source, Expand: true, ReadHeader: config.Server.TypeDetectionByHeader, Checker: d.user, @@ -88,26 +90,25 @@ func previewHandler(w http.ResponseWriter, r *http.Request, d *requestContext) ( if err != nil { return errToStatus(err), err } - cacheKey := previewCacheKey(fileInfo, previewSize) + cacheKey := previewCacheKey(response.RealPath, previewSize, fileInfo.ModTime) resizedImage, ok, err := fileCache.Load(r.Context(), cacheKey) if err != nil { return errToStatus(err), err } if !ok { - resizedImage, err = createPreview(imgSvc, fileCache, fileInfo, previewSize) + resizedImage, err = createPreview(imgSvc, fileCache, response, previewSize) if err != nil { return errToStatus(err), err } } w.Header().Set("Cache-Control", "private") - http.ServeContent(w, r, fileInfo.RealPath(), fileInfo.ModTime, bytes.NewReader(resizedImage)) - + http.ServeContent(w, r, response.RealPath, fileInfo.ModTime, bytes.NewReader(resizedImage)) return 0, nil } -func createPreview(imgSvc ImgService, fileCache FileCache, file *files.FileInfo, previewSize string) ([]byte, error) { - fd, err := os.Open(file.RealPath()) +func createPreview(imgSvc ImgService, fileCache FileCache, file files.ExtendedFileInfo, previewSize string) ([]byte, error) { + fd, err := os.Open(file.RealPath) if err != nil { return nil, err } @@ -138,7 +139,7 @@ func createPreview(imgSvc ImgService, fileCache FileCache, file *files.FileInfo, } go func() { - cacheKey := previewCacheKey(file, previewSize) + cacheKey := previewCacheKey(file.RealPath, previewSize, file.FileInfo.ModTime) if err := fileCache.Store(context.Background(), cacheKey, buf.Bytes()); err != nil { fmt.Printf("failed to cache resized image: %v", err) } @@ -148,12 +149,13 @@ func createPreview(imgSvc ImgService, fileCache FileCache, file *files.FileInfo, } // Generates a cache key for the preview image -func previewCacheKey(f *files.FileInfo, previewSize string) string { - return fmt.Sprintf("%x%x%x", f.RealPath(), f.ModTime.Unix(), previewSize) +func previewCacheKey(realPath, previewSize string, modTime time.Time) string { + return fmt.Sprintf("%x%x%x", realPath, modTime.Unix(), previewSize) } func rawFileHandler(w http.ResponseWriter, r *http.Request, file *files.FileInfo) (int, error) { - realPath, _, _ := files.GetRealPath(file.Path) + idx := files.GetIndex("default") + realPath, _, _ := idx.GetRealPath(file.Path) fd, err := os.Open(realPath) if err != nil { return http.StatusInternalServerError, err diff --git a/backend/http/public.go b/backend/http/public.go index f68192a3..f2d0d8e3 100644 --- a/backend/http/public.go +++ b/backend/http/public.go @@ -7,7 +7,6 @@ import ( "strings" "github.com/gtsteffaniak/filebrowser/backend/files" - "github.com/gtsteffaniak/filebrowser/backend/settings" "github.com/gtsteffaniak/filebrowser/backend/users" _ "github.com/gtsteffaniak/filebrowser/backend/swagger/docs" @@ -18,7 +17,7 @@ func publicShareHandler(w http.ResponseWriter, r *http.Request, d *requestContex if !ok { return http.StatusInternalServerError, fmt.Errorf("failed to assert type *files.FileInfo") } - file.Path = strings.TrimPrefix(file.Path, settings.Config.Server.Root) + file.Path = strings.TrimPrefix(file.Path, files.RootPaths["default"]) return renderJSON(w, r, file) } diff --git a/backend/http/raw.go b/backend/http/raw.go index 5116f0c8..4571c023 100644 --- a/backend/http/raw.go +++ b/backend/http/raw.go @@ -55,7 +55,8 @@ func rawHandler(w http.ResponseWriter, r *http.Request, d *requestContext) (int, } func addFile(path string, d *requestContext, tarWriter *tar.Writer, zipWriter *zip.Writer) error { - realPath, _, _ := files.GetRealPath(d.user.Scope, path) + idx := files.GetIndex("default") + realPath, _, _ := idx.GetRealPath(d.user.Scope, path) if !d.user.Check(realPath) { return nil } @@ -143,7 +144,8 @@ func addSingleFile(realPath, archivePath string, zipWriter *zip.Writer, tarWrite func rawFilesHandler(w http.ResponseWriter, r *http.Request, d *requestContext, fileList []string) (int, error) { filePath := fileList[0] fileName := filepath.Base(filePath) - realPath, isDir, err := files.GetRealPath(d.user.Scope, filePath) + idx := files.GetIndex("default") + realPath, isDir, err := idx.GetRealPath(d.user.Scope, filePath) if err != nil { return http.StatusInternalServerError, err } diff --git a/backend/http/resource.go b/backend/http/resource.go index cf51d611..5d5a324d 100644 --- a/backend/http/resource.go +++ b/backend/http/resource.go @@ -25,6 +25,7 @@ import ( // @Accept json // @Produce json // @Param path query string true "Path to the resource" +// @Param source query string false "Source name for the desired source, default is used if not provided" // @Param source query string false "Name for the desired source, default is used if not provided" // @Param content query string false "Include file content if true" // @Param checksum query string false "Optional checksum validation" @@ -33,8 +34,11 @@ import ( // @Failure 500 {object} map[string]string "Internal server error" // @Router /api/resources [get] func resourceGetHandler(w http.ResponseWriter, r *http.Request, d *requestContext) (int, error) { - // TODO source := r.URL.Query().Get("source") encodedPath := r.URL.Query().Get("path") + source := r.URL.Query().Get("source") + if source == "" { + source = "default" + } // Decode the URL-encoded path path, err := url.QueryUnescape(encodedPath) if err != nil { @@ -43,6 +47,7 @@ func resourceGetHandler(w http.ResponseWriter, r *http.Request, d *requestContex fileInfo, err := files.FileInfoFaster(files.FileOptions{ Path: filepath.Join(d.user.Scope, path), Modify: d.user.Perm.Modify, + Source: source, Expand: true, ReadHeader: config.Server.TypeDetectionByHeader, Checker: d.user, @@ -74,6 +79,7 @@ func resourceGetHandler(w http.ResponseWriter, r *http.Request, d *requestContex // @Accept json // @Produce json // @Param path query string true "Path to the resource" +// @Param source query string false "Source name for the desired source, default is used if not provided" // @Param source query string false "Name for the desired source, default is used if not provided" // @Success 200 "Resource deleted successfully" // @Failure 403 {object} map[string]string "Forbidden" @@ -83,6 +89,10 @@ func resourceGetHandler(w http.ResponseWriter, r *http.Request, d *requestContex func resourceDeleteHandler(w http.ResponseWriter, r *http.Request, d *requestContext) (int, error) { // TODO source := r.URL.Query().Get("source") encodedPath := r.URL.Query().Get("path") + source := r.URL.Query().Get("source") + if source == "" { + source = "default" + } // Decode the URL-encoded path path, err := url.QueryUnescape(encodedPath) if err != nil { @@ -91,13 +101,9 @@ func resourceDeleteHandler(w http.ResponseWriter, r *http.Request, d *requestCon if path == "/" || !d.user.Perm.Delete { return http.StatusForbidden, nil } - realPath, isDir, err := files.GetRealPath(d.user.Scope, path) - if err != nil { - return http.StatusNotFound, err - } fileOpts := files.FileOptions{ - Path: realPath, - IsDir: isDir, + Path: filepath.Join(d.user.Scope, path), + Source: source, Modify: d.user.Perm.Modify, Expand: false, ReadHeader: config.Server.TypeDetectionByHeader, @@ -109,12 +115,12 @@ func resourceDeleteHandler(w http.ResponseWriter, r *http.Request, d *requestCon } // delete thumbnails - err = delThumbs(r.Context(), fileCache, fileInfo.FileInfo) + err = delThumbs(r.Context(), fileCache, fileInfo) if err != nil { return errToStatus(err), err } - err = files.DeleteFiles(realPath, fileOpts) + err = files.DeleteFiles(source, fileInfo.RealPath, filepath.Dir(fileInfo.RealPath)) if err != nil { return errToStatus(err), err } @@ -129,6 +135,7 @@ func resourceDeleteHandler(w http.ResponseWriter, r *http.Request, d *requestCon // @Accept json // @Produce json // @Param path query string true "Path to the resource" +// @Param source query string false "Source name for the desired source, default is used if not provided" // @Param source query string false "Name for the desired source, default is used if not provided" // @Param override query bool false "Override existing file if true" // @Success 200 "Resource created successfully" @@ -138,8 +145,11 @@ func resourceDeleteHandler(w http.ResponseWriter, r *http.Request, d *requestCon // @Failure 500 {object} map[string]string "Internal server error" // @Router /api/resources [post] func resourcePostHandler(w http.ResponseWriter, r *http.Request, d *requestContext) (int, error) { - // TODO source := r.URL.Query().Get("source") encodedPath := r.URL.Query().Get("path") + source := r.URL.Query().Get("source") + if source == "" { + source = "default" + } // Decode the URL-encoded path path, err := url.QueryUnescape(encodedPath) if err != nil { @@ -150,6 +160,7 @@ func resourcePostHandler(w http.ResponseWriter, r *http.Request, d *requestConte } fileOpts := files.FileOptions{ Path: filepath.Join(d.user.Scope, path), + Source: source, Modify: d.user.Perm.Modify, Expand: false, Checker: d.user, @@ -173,7 +184,7 @@ func resourcePostHandler(w http.ResponseWriter, r *http.Request, d *requestConte return http.StatusForbidden, nil } - err = delThumbs(r.Context(), fileCache, fileInfo.FileInfo) + err = delThumbs(r.Context(), fileCache, fileInfo) if err != nil { return errToStatus(err), err } @@ -193,6 +204,7 @@ func resourcePostHandler(w http.ResponseWriter, r *http.Request, d *requestConte // @Accept json // @Produce json // @Param path query string true "Path to the resource" +// @Param source query string false "Source name for the desired source, default is used if not provided" // @Param source query string false "Name for the desired source, default is used if not provided" // @Success 200 "Resource updated successfully" // @Failure 403 {object} map[string]string "Forbidden" @@ -201,9 +213,13 @@ func resourcePostHandler(w http.ResponseWriter, r *http.Request, d *requestConte // @Failure 500 {object} map[string]string "Internal server error" // @Router /api/resources [put] func resourcePutHandler(w http.ResponseWriter, r *http.Request, d *requestContext) (int, error) { - // TODO source := r.URL.Query().Get("source") - // TODO source := r.URL.Query().Get("source") + source := r.URL.Query().Get("source") + if source == "" { + source = "default" + } + encodedPath := r.URL.Query().Get("path") + // Decode the URL-encoded path path, err := url.QueryUnescape(encodedPath) if err != nil { @@ -218,13 +234,9 @@ func resourcePutHandler(w http.ResponseWriter, r *http.Request, d *requestContex return http.StatusMethodNotAllowed, nil } - realPath, isDir, err := files.GetRealPath(d.user.Scope, path) - if err != nil { - return http.StatusNotFound, err - } fileOpts := files.FileOptions{ - Path: realPath, - IsDir: isDir, + Path: filepath.Join(d.user.Scope, path), + Source: source, Modify: d.user.Perm.Modify, Expand: false, ReadHeader: config.Server.TypeDetectionByHeader, @@ -241,6 +253,7 @@ func resourcePutHandler(w http.ResponseWriter, r *http.Request, d *requestContex // @Accept json // @Produce json // @Param from query string true "Path from resource" +// @Param source query string false "Source name for the desired source, default is used if not provided" // @Param destination query string true "Destination path for the resource" // @Param action query string true "Action to perform (copy, rename)" // @Param overwrite query bool false "Overwrite if destination exists" @@ -254,6 +267,10 @@ func resourcePutHandler(w http.ResponseWriter, r *http.Request, d *requestContex func resourcePatchHandler(w http.ResponseWriter, r *http.Request, d *requestContext) (int, error) { // TODO source := r.URL.Query().Get("source") action := r.URL.Query().Get("action") + source := r.URL.Query().Get("source") + if source == "" { + source = "default" + } encodedFrom := r.URL.Query().Get("from") // Decode the URL-encoded path src, err := url.QueryUnescape(encodedFrom) @@ -272,13 +289,14 @@ func resourcePatchHandler(w http.ResponseWriter, r *http.Request, d *requestCont return http.StatusForbidden, nil } + idx := files.GetIndex(source) // check target dir exists - parentDir, _, err := files.GetRealPath(d.user.Scope, filepath.Dir(dst)) + parentDir, _, err := idx.GetRealPath(d.user.Scope, filepath.Dir(dst)) if err != nil { return http.StatusNotFound, err } realDest := parentDir + "/" + filepath.Base(dst) - realSrc, isSrcDir, err := files.GetRealPath(d.user.Scope, src) + realSrc, isSrcDir, err := idx.GetRealPath(d.user.Scope, src) if err != nil { return http.StatusNotFound, err } @@ -292,7 +310,7 @@ func resourcePatchHandler(w http.ResponseWriter, r *http.Request, d *requestCont return http.StatusForbidden, nil } err = d.RunHook(func() error { - return patchAction(r.Context(), action, realSrc, realDest, d, fileCache, isSrcDir) + return patchAction(r.Context(), action, realSrc, realDest, d, fileCache, isSrcDir, source) }, action, realSrc, realDest, d.user) return errToStatus(err), err @@ -314,20 +332,20 @@ func addVersionSuffix(source string) string { return source } -func delThumbs(ctx context.Context, fileCache FileCache, file *files.FileInfo) error { - if err := fileCache.Delete(ctx, previewCacheKey(file, "small")); err != nil { +func delThumbs(ctx context.Context, fileCache FileCache, file files.ExtendedFileInfo) error { + if err := fileCache.Delete(ctx, previewCacheKey(file.RealPath, "small", file.FileInfo.ModTime)); err != nil { return err } return nil } -func patchAction(ctx context.Context, action, src, dst string, d *requestContext, fileCache FileCache, isSrcDir bool) error { +func patchAction(ctx context.Context, action, src, dst string, d *requestContext, fileCache FileCache, isSrcDir bool, index string) error { switch action { case "copy": if !d.user.Perm.Create { return errors.ErrPermissionDenied } - err := files.CopyResource(src, dst, isSrcDir) + err := files.CopyResource(index, src, dst, isSrcDir) return err case "rename", "move": @@ -336,6 +354,7 @@ func patchAction(ctx context.Context, action, src, dst string, d *requestContext } fileInfo, err := files.FileInfoFaster(files.FileOptions{ Path: src, + Source: index, IsDir: isSrcDir, Modify: d.user.Perm.Modify, Expand: false, @@ -347,11 +366,11 @@ func patchAction(ctx context.Context, action, src, dst string, d *requestContext } // delete thumbnails - err = delThumbs(ctx, fileCache, fileInfo.FileInfo) + err = delThumbs(ctx, fileCache, fileInfo) if err != nil { return err } - return files.MoveResource(src, dst, isSrcDir) + return files.MoveResource(index, src, dst, isSrcDir) default: return fmt.Errorf("unsupported action %s: %w", action, errors.ErrInvalidRequestParams) } @@ -368,7 +387,7 @@ type DiskUsageResponse struct { // @Tags Resources // @Accept json // @Produce json -// @Param source query string false "Name for the desired source, default is used if not provided" +// @Param source query string false "Source name for the desired source, default is used if not provided" // @Success 200 {object} DiskUsageResponse "Disk usage details" // @Failure 404 {object} map[string]string "Directory not found" // @Failure 500 {object} map[string]string "Internal server error" @@ -376,22 +395,19 @@ type DiskUsageResponse struct { func diskUsage(w http.ResponseWriter, r *http.Request, d *requestContext) (int, error) { source := r.URL.Query().Get("source") if source == "" { - source = "/" + source = "default" } - value, ok := utils.DiskUsageCache.Get(source).(DiskUsageResponse) if ok { return renderJSON(w, r, &value) } - fPath, isDir, err := files.GetRealPath(d.user.Scope, source) - if err != nil { - return errToStatus(err), err + rootPath, ok := files.RootPaths[source] + if !ok { + return 400, fmt.Errorf("bad source path provided: %v", source) } - if !isDir { - return http.StatusNotFound, fmt.Errorf("not a directory: %s", source) - } - usage, err := disk.UsageWithContext(r.Context(), fPath) + + usage, err := disk.UsageWithContext(r.Context(), rootPath) if err != nil { return errToStatus(err), err } @@ -405,11 +421,15 @@ func diskUsage(w http.ResponseWriter, r *http.Request, d *requestContext) (int, func inspectIndex(w http.ResponseWriter, r *http.Request) { encodedPath := r.URL.Query().Get("path") + source := r.URL.Query().Get("source") + if source == "" { + source = "default" + } // Decode the URL-encoded path path, _ := url.QueryUnescape(encodedPath) - isDir := r.URL.Query().Get("isDir") == "true" - index := files.GetIndex(config.Server.Root) - info, _ := index.GetReducedMetadata(path, isDir) + isNotDir := r.URL.Query().Get("isDir") == "false" // default to isDir true + index := files.GetIndex(source) + info, _ := index.GetReducedMetadata(path, !isNotDir) renderJSON(w, r, info) // nolint:errcheck } diff --git a/backend/http/search.go b/backend/http/search.go index 22eb5942..603137f2 100644 --- a/backend/http/search.go +++ b/backend/http/search.go @@ -5,7 +5,6 @@ import ( "strings" "github.com/gtsteffaniak/filebrowser/backend/files" - "github.com/gtsteffaniak/filebrowser/backend/settings" ) // searchHandler handles search requests for files based on the provided query. @@ -54,11 +53,15 @@ import ( // @Router /api/search [get] func searchHandler(w http.ResponseWriter, r *http.Request, d *requestContext) (int, error) { query := r.URL.Query().Get("query") + source := r.URL.Query().Get("source") + if source == "" { + source = "default" + } searchScope := strings.TrimPrefix(r.URL.Query().Get("scope"), ".") searchScope = strings.TrimPrefix(searchScope, "/") // Retrieve the User-Agent and X-Auth headers from the request sessionId := r.Header.Get("SessionId") - index := files.GetIndex(settings.Config.Server.Root) + index := files.GetIndex(source) userScope := strings.TrimPrefix(d.user.Scope, ".") combinedScope := strings.TrimPrefix(userScope+"/"+searchScope, "/") diff --git a/backend/http/share.go b/backend/http/share.go index ab9e6684..1dc9f1d0 100644 --- a/backend/http/share.go +++ b/backend/http/share.go @@ -68,6 +68,7 @@ func shareGetsHandler(w http.ResponseWriter, r *http.Request, d *requestContext) if err == errors.ErrNotExist { return renderJSON(w, r, []*share.Link{}) } + if err != nil { return http.StatusInternalServerError, fmt.Errorf("error getting share info from server") } diff --git a/backend/http/static.go b/backend/http/static.go index 4fea1839..f28058ab 100644 --- a/backend/http/static.go +++ b/backend/http/static.go @@ -44,7 +44,7 @@ func handleWithStaticData(w http.ResponseWriter, r *http.Request, file, contentT data := map[string]interface{}{ "Name": config.Frontend.Name, - "DisableExternal": config.Frontend.DisableExternal, + "DisableExternal": config.Frontend.DisableDefaultLinks, "DisableUsedPercentage": config.Frontend.DisableUsedPercentage, "darkMode": settings.Config.UserDefaults.DarkMode, "Color": config.Frontend.Color, @@ -62,6 +62,7 @@ func handleWithStaticData(w http.ResponseWriter, r *http.Request, file, contentT "ResizePreview": config.Server.ResizePreview, "EnableExec": config.Server.EnableExec, "ReCaptchaHost": config.Auth.Recaptcha.Host, + "ExternalLinks": config.Frontend.ExternalLinks, } if config.Frontend.Files != "" { diff --git a/backend/http/users.go b/backend/http/users.go index d42709e5..134c1738 100644 --- a/backend/http/users.go +++ b/backend/http/users.go @@ -55,26 +55,29 @@ func userGetHandler(w http.ResponseWriter, r *http.Request, d *requestContext) ( if givenUserIdString == "self" { givenUserId = d.user.ID } else if givenUserIdString == "" { - if !d.user.Perm.Admin { - return http.StatusForbidden, nil - } - users, err := store.Users.Gets(config.Server.Root) + + userList, err := store.Users.Gets(files.RootPaths["default"]) if err != nil { return http.StatusInternalServerError, err } - for _, u := range users { + selfUserList := []*users.User{} + for _, u := range userList { u.Password = "" - } - for _, u := range users { u.ApiKeys = nil + if u.ID == d.user.ID { + selfUserList = append(selfUserList, u) + } } - sort.Slice(users, func(i, j int) bool { - return users[i].ID < users[j].ID + sort.Slice(userList, func(i, j int) bool { + return userList[i].ID < userList[j].ID }) - return renderJSON(w, r, users) + if !d.user.Perm.Admin { + userList = selfUserList + } + return renderJSON(w, r, userList) } else { num, _ := strconv.ParseUint(givenUserIdString, 10, 32) givenUserId = uint(num) @@ -85,7 +88,7 @@ func userGetHandler(w http.ResponseWriter, r *http.Request, d *requestContext) ( } // Fetch the user details - u, err := store.Users.Get(config.Server.Root, givenUserId) + u, err := store.Users.Get(files.RootPaths["default"], givenUserId) if err == errors.ErrNotExist { return http.StatusNotFound, err } @@ -156,7 +159,8 @@ func usersPostHandler(w http.ResponseWriter, r *http.Request, d *requestContext) } // Validate the user's scope - _, _, err := files.GetRealPath(config.Server.Root, d.user.Scope) + idx := files.GetIndex("default") + _, _, err := idx.GetRealPath(d.user.Scope) if err != nil { return http.StatusBadRequest, err } @@ -214,7 +218,8 @@ func userPutHandler(w http.ResponseWriter, r *http.Request, d *requestContext) ( } // Validate the user's scope - _, _, err := files.GetRealPath(config.Server.Root, d.user.Scope) + idx := files.GetIndex("default") + _, _, err := idx.GetRealPath(d.user.Scope) if err != nil { return http.StatusBadRequest, err } diff --git a/backend/runner/runner.go b/backend/runner/runner.go index c70f30bb..24149043 100644 --- a/backend/runner/runner.go +++ b/backend/runner/runner.go @@ -20,8 +20,9 @@ type Runner struct { // RunHook runs the hooks for the before and after event. func (r *Runner) RunHook(fn func() error, evt, path, dst string, user *users.User) error { - path, _, _ = files.GetRealPath(user.Scope, path) - dst, _, _ = files.GetRealPath(user.Scope, dst) + idx := files.GetIndex("default") + path, _, _ = idx.GetRealPath(user.Scope, path) + dst, _, _ = idx.GetRealPath(user.Scope, dst) if r.Enabled { if val, ok := r.Commands["before_"+evt]; ok { diff --git a/backend/settings/config.go b/backend/settings/config.go index e8f10d2f..2cbc1048 100644 --- a/backend/settings/config.go +++ b/backend/settings/config.go @@ -1,6 +1,7 @@ package settings import ( + "fmt" "log" "os" "path/filepath" @@ -8,6 +9,7 @@ import ( "github.com/goccy/go-yaml" "github.com/gtsteffaniak/filebrowser/backend/users" + "github.com/gtsteffaniak/filebrowser/backend/version" ) var Config Settings @@ -21,21 +23,50 @@ func Initialize(configFile string) { } Config.UserDefaults.Perm = Config.UserDefaults.Permissions // Convert relative path to absolute path - realRoot, err := filepath.Abs(Config.Server.Root) - if err != nil { - log.Fatalf("Error getting root path: %v", err) + if len(Config.Server.Sources) > 0 { + // TODO allow multipe sources not named default + for _, source := range Config.Server.Sources { + realPath, err := filepath.Abs(source.Path) + if err != nil { + log.Fatalf("Error getting source path: %v", err) + } + source.Path = realPath + source.Name = "default" // Modify the local copy of the map value + Config.Server.Sources["default"] = source // Assign the modified value back to the map + } + } else { + realPath, err := filepath.Abs(Config.Server.Root) + if err != nil { + log.Fatalf("Error getting source path: %v", err) + } + Config.Server.Sources = map[string]Source{ + "default": { + Name: "default", + Path: realPath, + }, + } } - _, err = os.Stat(realRoot) - if err != nil { - log.Fatalf("ERROR: Configured Root Path does not exist! %v", err) - } - Config.Server.Root = realRoot baseurl := strings.Trim(Config.Server.BaseURL, "/") if baseurl == "" { Config.Server.BaseURL = "/" } else { Config.Server.BaseURL = "/" + baseurl + "/" } + if !Config.Frontend.DisableDefaultLinks { + Config.Frontend.ExternalLinks = append(Config.Frontend.ExternalLinks, ExternalLink{ + Text: "FileBrowser Quantum", + Url: "https://github.com/gtsteffaniak/filebrowser", + }) + Config.Frontend.ExternalLinks = append(Config.Frontend.ExternalLinks, ExternalLink{ + Text: fmt.Sprintf("(%v)", version.Version), + Title: version.CommitSHA, + Url: "https://github.com/gtsteffaniak/filebrowser/releases/", + }) + Config.Frontend.ExternalLinks = append(Config.Frontend.ExternalLinks, ExternalLink{ + Text: "Help", + Url: "https://github.com/gtsteffaniak/filebrowser/wiki", + }) + } } func loadConfigFile(configFile string) []byte { @@ -72,7 +103,6 @@ func setDefaults() Settings { Database: "database.db", Log: "stdout", Root: "/srv", - Indexing: true, }, Auth: Auth{ TokenExpirationTime: "2h", diff --git a/backend/settings/settings_test.go b/backend/settings/settings_test.go index 26f30b7c..3a3da0bb 100644 --- a/backend/settings/settings_test.go +++ b/backend/settings/settings_test.go @@ -41,14 +41,13 @@ func TestConfigLoadSpecificValues(t *testing.T) { }{ {"Auth.Method", Config.Auth.Method, newConfig.Auth.Method}, {"Auth.Method", Config.Auth.Method, newConfig.Auth.Method}, - {"Frontend.disableExternal", Config.Frontend.DisableExternal, newConfig.Frontend.DisableExternal}, {"UserDefaults.HideDotfiles", Config.UserDefaults.HideDotfiles, newConfig.UserDefaults.HideDotfiles}, {"Server.Database", Config.Server.Database, newConfig.Server.Database}, } for _, tc := range testCases { if tc.globalVal == tc.newVal { - t.Errorf("Differences should have been found:\n\tConfig.%s: %v \n\tSetConfig: %v \n", tc.fieldName, tc.globalVal, tc.newVal) + t.Errorf("Differences should have been found:\nConfig.%s: %v \nSetConfig: %v \n", tc.fieldName, tc.globalVal, tc.newVal) } } } diff --git a/backend/settings/structs.go b/backend/settings/structs.go index 16377433..59db6722 100644 --- a/backend/settings/structs.go +++ b/backend/settings/structs.go @@ -35,33 +35,62 @@ type Recaptcha struct { } type Server struct { - IndexingInterval uint32 `json:"indexingInterval"` - NumImageProcessors int `json:"numImageProcessors"` - Socket string `json:"socket"` - TLSKey string `json:"tlsKey"` - TLSCert string `json:"tlsCert"` - EnableThumbnails bool `json:"enableThumbnails"` - ResizePreview bool `json:"resizePreview"` - EnableExec bool `json:"enableExec"` - TypeDetectionByHeader bool `json:"typeDetectionByHeader"` - AuthHook string `json:"authHook"` - Port int `json:"port"` - BaseURL string `json:"baseURL"` - Address string `json:"address"` - Log string `json:"log"` - Database string `json:"database"` - Root string `json:"root"` - UserHomeBasePath string `json:"userHomeBasePath"` - CreateUserDir bool `json:"createUserDir"` - Indexing bool `json:"indexing"` + NumImageProcessors int `json:"numImageProcessors"` + Socket string `json:"socket"` + TLSKey string `json:"tlsKey"` + TLSCert string `json:"tlsCert"` + EnableThumbnails bool `json:"enableThumbnails"` + ResizePreview bool `json:"resizePreview"` + EnableExec bool `json:"enableExec"` + TypeDetectionByHeader bool `json:"typeDetectionByHeader"` + AuthHook string `json:"authHook"` + Port int `json:"port"` + BaseURL string `json:"baseURL"` + Address string `json:"address"` + Log string `json:"log"` + Database string `json:"database"` + Root string `json:"root"` + UserHomeBasePath string `json:"userHomeBasePath"` + CreateUserDir bool `json:"createUserDir"` + Sources map[string]Source `json:"sources"` +} + +type Source struct { + Path string `json:"path"` + Name string + Config IndexConfig `json:"config"` +} + +type IndexConfig struct { + IndexingInterval uint32 `json:"indexingInterval"` + Disabled bool `json:"disabled"` + MaxWatchers int `json:"maxWatchers"` + NeverWatch []string `json:"neverWatchPaths"` + IgnoreHidden bool `json:"ignoreHidden"` + IgnoreZeroSizeFolders bool `json:"ignoreZeroSizeFolders"` + Exclude IndexFilter `json:"exclude"` + Include IndexFilter `json:"include"` +} + +type IndexFilter struct { + Files []string `json:"files"` + Folders []string `json:"folders"` + FileEndsWith []string `json:"fileEndsWith"` } type Frontend struct { - Name string `json:"name"` - DisableExternal bool `json:"disableExternal"` - DisableUsedPercentage bool `json:"disableUsedPercentage"` - Files string `json:"files"` - Color string `json:"color"` + Name string `json:"name"` + DisableDefaultLinks bool `json:"disableDefaultLinks"` + DisableUsedPercentage bool `json:"disableUsedPercentage"` + Files string `json:"files"` + Color string `json:"color"` + ExternalLinks []ExternalLink `json:"externalLinks"` +} + +type ExternalLink struct { + Text string `json:"text"` + Title string `json:"title"` + Url string `json:"url"` } // UserDefaults is a type that holds the default values @@ -86,4 +115,5 @@ type UserDefaults struct { Commands []string `json:"commands,omitempty"` HideDotfiles bool `json:"hideDotfiles"` DateFormat bool `json:"dateFormat"` + ThemeColor string `json:"themeColor"` } diff --git a/backend/storage/bolt/share.go b/backend/storage/bolt/share.go index e054139b..c070169d 100644 --- a/backend/storage/bolt/share.go +++ b/backend/storage/bolt/share.go @@ -1,6 +1,9 @@ package bolt import ( + "log" + "time" + "github.com/asdine/storm/v3" "github.com/asdine/storm/v3/q" @@ -59,7 +62,20 @@ func (s shareBackend) Gets(path string, id uint) ([]*share.Link, error) { return v, errors.ErrNotExist } - return v, err + filteredList := []*share.Link{} + // automatically delete and clear expired shares + for i := range v { + if v[i].Expire < time.Now().Unix() { + err = s.Delete(v[i].PasswordHash) + if err != nil { + log.Println("expired share could not be deleted: ", err.Error()) + } + } else { + filteredList = append(filteredList, v[i]) + } + } + + return filteredList, err } func (s shareBackend) Save(l *share.Link) error { diff --git a/backend/storage/storage.go b/backend/storage/storage.go index 62eb6af1..f3731b18 100644 --- a/backend/storage/storage.go +++ b/backend/storage/storage.go @@ -116,14 +116,15 @@ func CreateUser(userInfo users.User, asAdmin bool) error { newUser.Perm = settings.AdminPerms() } // create new home directory - userHome, err := settings.Config.MakeUserDir(newUser.Username, newUser.Scope, settings.Config.Server.Root) + userHome, err := settings.Config.MakeUserDir(newUser.Username, newUser.Scope, files.RootPaths["default"]) if err != nil { log.Printf("create user: failed to mkdir user home dir: [%s]", userHome) return err } newUser.Scope = userHome log.Printf("user: %s, home dir: [%s].", newUser.Username, userHome) - _, _, err = files.GetRealPath(settings.Config.Server.Root, newUser.Scope) + idx := files.GetIndex("default") + _, _, err = idx.GetRealPath(newUser.Scope) if err != nil { log.Println("user path is not valid", newUser.Scope) return nil diff --git a/backend/swagger/docs/docs.go b/backend/swagger/docs/docs.go index 4ae596d9..a20adf8b 100644 --- a/backend/swagger/docs/docs.go +++ b/backend/swagger/docs/docs.go @@ -199,6 +199,12 @@ const docTemplate = `{ "in": "query", "required": true }, + { + "type": "string", + "description": "Source name for the desired source, default is used if not provided", + "name": "source", + "in": "query" + }, { "type": "string", "description": "Name for the desired source, default is used if not provided", @@ -265,6 +271,12 @@ const docTemplate = `{ "in": "query", "required": true }, + { + "type": "string", + "description": "Source name for the desired source, default is used if not provided", + "name": "source", + "in": "query" + }, { "type": "string", "description": "Name for the desired source, default is used if not provided", @@ -334,6 +346,12 @@ const docTemplate = `{ "in": "query", "required": true }, + { + "type": "string", + "description": "Source name for the desired source, default is used if not provided", + "name": "source", + "in": "query" + }, { "type": "string", "description": "Name for the desired source, default is used if not provided", @@ -409,6 +427,12 @@ const docTemplate = `{ "in": "query", "required": true }, + { + "type": "string", + "description": "Source name for the desired source, default is used if not provided", + "name": "source", + "in": "query" + }, { "type": "string", "description": "Name for the desired source, default is used if not provided", @@ -469,6 +493,12 @@ const docTemplate = `{ "in": "query", "required": true }, + { + "type": "string", + "description": "Source name for the desired source, default is used if not provided", + "name": "source", + "in": "query" + }, { "type": "string", "description": "Destination path for the resource", @@ -857,7 +887,7 @@ const docTemplate = `{ "parameters": [ { "type": "string", - "description": "Name for the desired source, default is used if not provided", + "description": "Source name for the desired source, default is used if not provided", "name": "source", "in": "query" } @@ -1260,18 +1290,38 @@ const docTemplate = `{ } } }, + "settings.ExternalLink": { + "type": "object", + "properties": { + "text": { + "type": "string" + }, + "title": { + "type": "string" + }, + "url": { + "type": "string" + } + } + }, "settings.Frontend": { "type": "object", "properties": { "color": { "type": "string" }, - "disableExternal": { + "disableDefaultLinks": { "type": "boolean" }, "disableUsedPercentage": { "type": "boolean" }, + "externalLinks": { + "type": "array", + "items": { + "$ref": "#/definitions/settings.ExternalLink" + } + }, "files": { "type": "string" }, @@ -1342,6 +1392,9 @@ const docTemplate = `{ "stickySidebar": { "type": "boolean" }, + "themeColor": { + "type": "string" + }, "viewMode": { "type": "string" } @@ -1542,6 +1595,9 @@ const docTemplate = `{ "stickySidebar": { "type": "boolean" }, + "themeColor": { + "type": "string" + }, "username": { "type": "string" }, diff --git a/backend/swagger/docs/swagger.json b/backend/swagger/docs/swagger.json index 438ec6d3..dbb1b77d 100644 --- a/backend/swagger/docs/swagger.json +++ b/backend/swagger/docs/swagger.json @@ -188,6 +188,12 @@ "in": "query", "required": true }, + { + "type": "string", + "description": "Source name for the desired source, default is used if not provided", + "name": "source", + "in": "query" + }, { "type": "string", "description": "Name for the desired source, default is used if not provided", @@ -254,6 +260,12 @@ "in": "query", "required": true }, + { + "type": "string", + "description": "Source name for the desired source, default is used if not provided", + "name": "source", + "in": "query" + }, { "type": "string", "description": "Name for the desired source, default is used if not provided", @@ -323,6 +335,12 @@ "in": "query", "required": true }, + { + "type": "string", + "description": "Source name for the desired source, default is used if not provided", + "name": "source", + "in": "query" + }, { "type": "string", "description": "Name for the desired source, default is used if not provided", @@ -398,6 +416,12 @@ "in": "query", "required": true }, + { + "type": "string", + "description": "Source name for the desired source, default is used if not provided", + "name": "source", + "in": "query" + }, { "type": "string", "description": "Name for the desired source, default is used if not provided", @@ -458,6 +482,12 @@ "in": "query", "required": true }, + { + "type": "string", + "description": "Source name for the desired source, default is used if not provided", + "name": "source", + "in": "query" + }, { "type": "string", "description": "Destination path for the resource", @@ -846,7 +876,7 @@ "parameters": [ { "type": "string", - "description": "Name for the desired source, default is used if not provided", + "description": "Source name for the desired source, default is used if not provided", "name": "source", "in": "query" } @@ -1249,18 +1279,38 @@ } } }, + "settings.ExternalLink": { + "type": "object", + "properties": { + "text": { + "type": "string" + }, + "title": { + "type": "string" + }, + "url": { + "type": "string" + } + } + }, "settings.Frontend": { "type": "object", "properties": { "color": { "type": "string" }, - "disableExternal": { + "disableDefaultLinks": { "type": "boolean" }, "disableUsedPercentage": { "type": "boolean" }, + "externalLinks": { + "type": "array", + "items": { + "$ref": "#/definitions/settings.ExternalLink" + } + }, "files": { "type": "string" }, @@ -1331,6 +1381,9 @@ "stickySidebar": { "type": "boolean" }, + "themeColor": { + "type": "string" + }, "viewMode": { "type": "string" } @@ -1531,6 +1584,9 @@ "stickySidebar": { "type": "boolean" }, + "themeColor": { + "type": "string" + }, "username": { "type": "string" }, diff --git a/backend/swagger/docs/swagger.yaml b/backend/swagger/docs/swagger.yaml index 25345ba6..0d1aaee9 100644 --- a/backend/swagger/docs/swagger.yaml +++ b/backend/swagger/docs/swagger.yaml @@ -79,14 +79,27 @@ definitions: userHomeBasePath: type: string type: object + settings.ExternalLink: + properties: + text: + type: string + title: + type: string + url: + type: string + type: object settings.Frontend: properties: color: type: string - disableExternal: + disableDefaultLinks: type: boolean disableUsedPercentage: type: boolean + externalLinks: + items: + $ref: '#/definitions/settings.ExternalLink' + type: array files: type: string name: @@ -133,6 +146,8 @@ definitions: type: object stickySidebar: type: boolean + themeColor: + type: string viewMode: type: string type: object @@ -267,6 +282,8 @@ definitions: $ref: '#/definitions/users.Sorting' stickySidebar: type: boolean + themeColor: + type: string username: type: string viewMode: @@ -398,6 +415,10 @@ paths: name: path required: true type: string + - description: Source name for the desired source, default is used if not provided + in: query + name: source + type: string - description: Name for the desired source, default is used if not provided in: query name: source @@ -439,6 +460,10 @@ paths: name: path required: true type: string + - description: Source name for the desired source, default is used if not provided + in: query + name: source + type: string - description: Name for the desired source, default is used if not provided in: query name: source @@ -483,6 +508,10 @@ paths: name: from required: true type: string + - description: Source name for the desired source, default is used if not provided + in: query + name: source + type: string - description: Destination path for the resource in: query name: destination @@ -544,6 +573,10 @@ paths: name: path required: true type: string + - description: Source name for the desired source, default is used if not provided + in: query + name: source + type: string - description: Name for the desired source, default is used if not provided in: query name: source @@ -594,6 +627,10 @@ paths: name: path required: true type: string + - description: Source name for the desired source, default is used if not provided + in: query + name: source + type: string - description: Name for the desired source, default is used if not provided in: query name: source @@ -842,7 +879,7 @@ paths: - application/json description: Returns the total and used disk space for a specified directory. parameters: - - description: Name for the desired source, default is used if not provided + - description: Source name for the desired source, default is used if not provided in: query name: source type: string diff --git a/backend/test_output.txt b/backend/test_output.txt deleted file mode 100644 index 42fde92b..00000000 --- a/backend/test_output.txt +++ /dev/null @@ -1,113 +0,0 @@ - - == Running tests == -/usr/local/go/bin/go -? github.com/gtsteffaniak/filebrowser [no test files] -? github.com/gtsteffaniak/filebrowser/auth [no test files] -? github.com/gtsteffaniak/filebrowser/cmd [no test files] -? github.com/gtsteffaniak/filebrowser/errors [no test files] -? github.com/gtsteffaniak/filebrowser/files [no test files] -=== RUN TestFileCache ---- PASS: TestFileCache (0.00s) -PASS -ok github.com/gtsteffaniak/filebrowser/diskcache (cached) -=== RUN TestCommonPrefix -=== RUN TestCommonPrefix/sub_folder -=== RUN TestCommonPrefix/relative_path -=== RUN TestCommonPrefix/no_common_path -=== RUN TestCommonPrefix/same_lvl ---- PASS: TestCommonPrefix (0.00s) - --- PASS: TestCommonPrefix/sub_folder (0.00s) - --- PASS: TestCommonPrefix/relative_path (0.00s) - --- PASS: TestCommonPrefix/no_common_path (0.00s) - --- PASS: TestCommonPrefix/same_lvl (0.00s) -PASS -ok github.com/gtsteffaniak/filebrowser/fileutils (cached) -? github.com/gtsteffaniak/filebrowser/settings [no test files] -? github.com/gtsteffaniak/filebrowser/share [no test files] -? github.com/gtsteffaniak/filebrowser/storage [no test files] -? github.com/gtsteffaniak/filebrowser/storage/bolt [no test files] -2023/09/02 13:07:17 Error opening YAML file: open filebrowser.yaml: no such file or directory -FAIL github.com/gtsteffaniak/filebrowser/http 0.008s -=== RUN TestService_Resize -=== RUN TestService_Resize/convert_to_png -=== RUN TestService_Resize/convert_to_tiff -=== RUN TestService_Resize/resize_bmp -=== RUN TestService_Resize/resize_with_medium_quality -=== RUN TestService_Resize/resize_with_low_quality -=== RUN TestService_Resize/get_thumbnail_from_file_with_APP0_JFIF -=== RUN TestService_Resize/fill_upscale -=== RUN TestService_Resize/fit_upscale -=== RUN TestService_Resize/convert_to_gif -=== RUN TestService_Resize/convert_to_bmp -=== RUN TestService_Resize/resize_tiff -=== RUN TestService_Resize/resize_with_high_quality -=== RUN TestService_Resize/fill_downscale -=== RUN TestService_Resize/keep_original_format -=== RUN TestService_Resize/convert_to_unknown -=== RUN TestService_Resize/get_thumbnail_from_file_without_APP0_JFIF -=== RUN TestService_Resize/resize_for_higher_quality_levels -=== RUN TestService_Resize/broken_file -=== RUN TestService_Resize/fit_downscale -=== RUN TestService_Resize/convert_to_jpeg -=== RUN TestService_Resize/resize_png -=== RUN TestService_Resize/resize_gif -=== RUN TestService_Resize/resize_with_unknown_quality -=== RUN TestService_Resize/resize_from_file_without_IFD1_thumbnail ---- PASS: TestService_Resize (1.36s) - --- PASS: TestService_Resize/convert_to_png (0.01s) - --- PASS: TestService_Resize/convert_to_tiff (0.01s) - --- PASS: TestService_Resize/resize_bmp (0.01s) - --- PASS: TestService_Resize/resize_with_medium_quality (0.01s) - --- PASS: TestService_Resize/resize_with_low_quality (0.01s) - --- PASS: TestService_Resize/get_thumbnail_from_file_with_APP0_JFIF (0.02s) - --- PASS: TestService_Resize/fill_upscale (0.01s) - --- PASS: TestService_Resize/fit_upscale (0.00s) - --- PASS: TestService_Resize/convert_to_gif (0.01s) - --- PASS: TestService_Resize/convert_to_bmp (0.01s) - --- PASS: TestService_Resize/resize_tiff (0.01s) - --- PASS: TestService_Resize/resize_with_high_quality (0.01s) - --- PASS: TestService_Resize/fill_downscale (0.01s) - --- PASS: TestService_Resize/keep_original_format (0.01s) - --- PASS: TestService_Resize/convert_to_unknown (0.01s) - --- PASS: TestService_Resize/get_thumbnail_from_file_without_APP0_JFIF (0.03s) - --- PASS: TestService_Resize/resize_for_higher_quality_levels (0.03s) - --- PASS: TestService_Resize/broken_file (0.00s) - --- PASS: TestService_Resize/fit_downscale (0.01s) - --- PASS: TestService_Resize/convert_to_jpeg (0.01s) - --- PASS: TestService_Resize/resize_png (0.02s) - --- PASS: TestService_Resize/resize_gif (0.02s) - --- PASS: TestService_Resize/resize_with_unknown_quality (0.01s) - --- PASS: TestService_Resize/resize_from_file_without_IFD1_thumbnail (1.09s) -=== RUN TestService_FormatFromExtension -=== RUN TestService_FormatFromExtension/gif -=== RUN TestService_FormatFromExtension/tiff -=== RUN TestService_FormatFromExtension/bmp -=== RUN TestService_FormatFromExtension/unknown -=== RUN TestService_FormatFromExtension/jpg -=== RUN TestService_FormatFromExtension/jpeg -=== RUN TestService_FormatFromExtension/png ---- PASS: TestService_FormatFromExtension (0.00s) - --- PASS: TestService_FormatFromExtension/gif (0.00s) - --- PASS: TestService_FormatFromExtension/tiff (0.00s) - --- PASS: TestService_FormatFromExtension/bmp (0.00s) - --- PASS: TestService_FormatFromExtension/unknown (0.00s) - --- PASS: TestService_FormatFromExtension/jpg (0.00s) - --- PASS: TestService_FormatFromExtension/jpeg (0.00s) - --- PASS: TestService_FormatFromExtension/png (0.00s) -PASS -ok github.com/gtsteffaniak/filebrowser/img (cached) -=== RUN TestMatchHidden ---- PASS: TestMatchHidden (0.00s) -PASS -ok github.com/gtsteffaniak/filebrowser/rules (cached) -2023/09/02 13:07:17 Error opening YAML file: open filebrowser.yaml: no such file or directory -FAIL github.com/gtsteffaniak/filebrowser/runner 0.007s -=== RUN TestParseSearch ---- PASS: TestParseSearch (0.00s) -PASS -ok github.com/gtsteffaniak/filebrowser/search (cached) -? github.com/gtsteffaniak/filebrowser/version [no test files] -testing: warning: no tests to run -PASS -ok github.com/gtsteffaniak/filebrowser/users (cached) [no tests to run] -FAIL diff --git a/backend/users/users.go b/backend/users/users.go index 49fb335c..78377e0a 100644 --- a/backend/users/users.go +++ b/backend/users/users.go @@ -55,6 +55,7 @@ type User struct { HideDotfiles bool `json:"hideDotfiles"` DateFormat bool `json:"dateFormat"` GallerySize int `json:"gallerySize"` + ThemeColor string `json:"themeColor"` } var PublicUser = User{ diff --git a/docs/configuration.md b/docs/configuration.md deleted file mode 100644 index b1601329..00000000 --- a/docs/configuration.md +++ /dev/null @@ -1,225 +0,0 @@ -# Configuration Help - -This document covers the available configuration options, their defaults, and how they affect the functionality of filebrowser. - -## All possible configurations - -Here is an expanded config file which includes all possible configurations: - -``` -server: - CreateUserDir: false - UserHomeBasePath: "" - indexing: true - numImageProcessors: 4 - socket: "" - tlsKey: "" - tlsCert: "" - enableThumbnails: false - resizePreview: true - typeDetectionByHeader: true - port: 80 - baseURL: "/" - address: "" - log: "stdout" - database: "/database/database.db" - root: "/srv" -auth: - adminUsername: admin - adminPassword: admin - recaptcha: - host: "" - key: "" - secret: "" - tokenExpirationTime: 2h - header: "" - method: json - signup: false -frontend: - name: "" - disableExternal: false - disableUsedPercentage: true - files: "" - color: "" -userDefaults: - settingsAllowed: true - darkMode: false - scope: "" - locale: "en" - viewMode: "" - singleClick: true - sorting: - by: "" - asc: true - permissions: - admin: true - execute: true - create: true - rename: true - modify: true - delete: true - share: true - download: true - hideDotfiles: false - dateFormat: false -``` - -Here are the defaults if nothing is set: - -``` -server: - enableThumbnails: true - enableExec: false - port: 80 - numImageProcessors: 4 - baseURL: "" - database: database.db - log: stdout - root: /srv -auth: - adminUsername: admin - adminPassword: admin - method: password - recaptcha: - host: "" -userDefaults: - settingsAllowed: true - darkMode: false - scope: "" - locale: "en" - scope: "." - lockPassword: false - hideDotfiles: true - permissions: - create: true - rename: true - modify: true - delete: true - share: true - download: true -``` - -## About each configuration - -# Configuration Settings Documentation - -## About each configuration - -### Server configuration settings - -- `indexingInterval`: This optional paramter disables smart indexing and specifies a time in minutes the system waits before checking for filesystem changes. See [indexing readme](indexing.md) for more information. - -- `indexing`: This enables or disables indexing. (Note: search will not work without indexing) Default: `true` - -- `numImageProcessors`: This is the number of image processors available. Default: `4` - -- `socket`: This is the socket configuration. - -- `tlsKey`: This is the TLS key configuration. - -- `tlsCert`: This is the TLS certificate configuration. - -- `enableThumbnails`: This boolean value determines whether thumbnails are enabled on ui. Default: `true` - -- `resizePreview`: This boolean value determines whether preview resizing is enabled. Default: `false` - -- `typeDetectionByHeader`: This boolean value determines whether type detection is based on headers. - -- `port`: This is the port number on which the server is running. Default: `80` - -- `baseURL`: This is the base URL for the server. Default: `""` - -- `address`: This is the server address configuration. Default: `0.0.0.0` - -- `log`: This specifies the log destination. Default: `stdout` - -- `database`: This is the database file path + filename that will be created if it does not already exist. If it exists, it will use the existing file. Default `database.db` - -- `root`: This is the root directory path. Default: `/srv` - -- `CreateUserDir`: Boolean to create user directory on user creation. Default: `false` - -- `UserHomeBasePath`: String to define user home directory base path. Default: `""` - -### Auth configuration settings - -- `recaptcha`: - - - `host`: This is the host for reCAPTCHA. - - - `key`: This is the reCAPTCHA key. - - - `secret`: This is the reCAPTCHA secret. - -- `header`: This is the authentication header. - -- `method`: This is the authentication method used (e.g., "json"). Possible values: - - `password` - username and password - - `hook` - hook authentication - - `proxy` - proxy authentication - - `oath` - oath authentication - - `noauth` - no authentication/login required. - -- `Signup`: This boolean value indicates whether user signup is enabled on the login page. NOTE: Be mindful of `userDefaults` settings if enabled. Default: `false` - -- `AdminUsername`: This is the username of the admin user. Default: `admin` - -- `AdminPassword`: This is the password of the admin user. Default: `admin` - -### Frontend configuration settings - -- `name`: This is the name of the frontend. - -- `disableExternal`: This boolean value determines whether external access is disabled. - -- `disableUsedPercentage`: This boolean value determines whether used percentage is disabled. - -- `files`: This is the files configuration. - -- `theme`: This is the theme configuration. - -- `color`: This is the color configuration. - -### UserDefaults configuration settings - -- `darkMode`: Determines whether dark mode is enabled for the user (`true` or `false`) - -- `settingsAllowed`: Determines whether settings page is enabled for the user (`true` or `false`) - -- `scope`: This is a scope of the permissions, "." or "./" means all directories, "./downloads" would mean only the downloads folder. - -- `locale`: String locale configuration. Default: `en` - -- `viewMode`: This is the view mode configuration. Possible values: `normal`, `compact`, `list`, and `gallery`. default: `normal` - -- `singleClick`: This boolean value determines whether single-click is enabled. (`true` or `false`) - -- `sorting`: - - - `by`: This is the sorting method used (e.g., "asc"). - - - `asc`: This boolean value determines the sorting order is ascending or descending. (`true` or `false`) - -- `permissions`: - - - `admin`: This boolean value determines whether admin permissions are granted. - - - `execute`: This boolean value determines whether execute permissions are granted. - - - `create`: This boolean value determines whether create permissions are granted. - - - `rename`: This boolean value determines whether rename permissions are granted. - - - `modify`: This boolean value determines whether modify permissions are granted. - - - `delete`: This boolean value determines whether delete permissions are granted. - - - `share`: This boolean value determines whether share permissions are granted. - - - `download`: This boolean value determines whether download permissions are granted. - - - `api`: Ability to create and manage API keys. - -- `hideDotfiles`: This boolean value determines whether dotfiles are hidden. (`true` or `false`) - -- `dateFormat`: This boolean value determines whether date formatting is enabled. (`true` or `false`) diff --git a/docs/contributing.md b/docs/contributing.md deleted file mode 100644 index df0e32fd..00000000 --- a/docs/contributing.md +++ /dev/null @@ -1,13 +0,0 @@ -# Contributing Guide - -If you would like to contribute, please open a pull request against main or the latest `dev_` branch thats currently in progress. - -A PR is required to have: - -1. A clear description about why it was opened -2. A short title that best describes the issue. -3. Test evidence for anything that is not self evident or covered by unit tests. - -Unit tests should be updated to pass before merging. So, the best way to handle this is to create a fork and test your changes there, then merge to this repo. You can also create a draft pull request if it is not fully ready. - -Please don't hesitate to open an issue for any ideas you have, but cannot contribute directly for whatever reason. \ No newline at end of file diff --git a/docs/indexing.md b/docs/indexing.md deleted file mode 100644 index 22444a38..00000000 --- a/docs/indexing.md +++ /dev/null @@ -1,189 +0,0 @@ -# About Indexing on FileBrowser Quantum - -The most significant feature is the index, this document intends to demystify how it works so you can understand how to ensure your index closely matches the current state of your filesystem. - -## How does the index work? - -The approach used by this repo includes filesystem watchers that periodically scan the directory tree for changes. By default, this uses a smart scan strategy, but you can also configure a set interval in your config file. - -The `scan interval` is the break time between scans and does not include the time a scan takes. A typical scan can vary dramatically, but here are some expectations for SSD-based disks: - -| # folders | # files | time to index | memory usage (RAM) | -|---|---|---|---| -| 10,000 | 10,000 | ~ 0-5 seconds | 15 MB | -| 2,000 | 250,000 | ~ 0-5 seconds | 300 MB | -| 50,000 | 50,000 | ~ 5-30 seconds | 150 MB | -| 250,000 | 10,000 | ~ 2-5 minutes | 300 MB | -| 500,000 | 500,000 | ~ 5+ minutes | 500+ MB | - -### Smart Scanning - -1. There is a floating `smart scan interval` that ranges from **1 minute - 4 hours** depending on the complexity of your filesystem -2. The smart interval changes based on how often it discovers changed files: - - ``` - // Schedule in minutes - var scanSchedule = []time.Duration{ - 5 * time.Minute, // 5 minute quick scan & 25 minutes for a full scan - 10 * time.Minute, - 20 * time.Minute, // [3] element is 20 minutes, reset anchor for full scan - 40 * time.Minute, - 1 * time.Hour, - 2 * time.Hour, - 3 * time.Hour, - 4 * time.Hour, // 4 hours for quick scan & 20 hours for a full scan - } - ``` -3. The `smart scan interval` performs a `quick scan` 4 times in a row, followed by a 5th `full scan` which completely rebuilds the index. - - A `quick scan` is limited to detecting directory changes, but is 10x faster than a full scan. Here is what a quick scan can see: - 1. New files or folders created. - 2. Files or folders deleted. - 3. Renaming of files or folders. - - A quick scan **cannot** detect when a file has been updated, for example when you save a file and the size increases. - - A `full scan` is a complete re-indexing. This is always more disk and computationally intense but will capture individual file changes. -4. The `smart scan interval` changes based on several things. A `simple` complexity enables scans every 1 minute if changes happen frequently and a maximum full scan interval of every 100 minutes. `high` complexity indicates a minimum scanning interval of 10 minutes. - - **under 10,000 folders** or **Under 3 seconds** to index is awlays considered `simple` complexity. - - **more than 500,000 folders** or **Over 2 minutes** to index is always considered `high` complexity. - -### Manual Scanning Interval - -If you don't like the behavior of smart scanning, you can configure set intervals instead by setting `indexingInterval` to a number greater than 0. This will make FileBrowser Quantum always scan at the given interval in minutes. - -The scan behavior is still 4 quick scans at a given interval, followed by a 5th full scan. - -### System requirements - -You can expect FileBrowser Quantum to use 100 MB of RAM for a typical installation. If you have many files and folders then the requirement could climb to multiple Gigabytes. Please monitor your system on the first run to know your specific requirements. - -### Why does FileBrowser Quantum index the way it does? - -The smart indexing method uses filesystem scanners because it allows a low-footprint design that can cater to individual filesystem complexity. There are a few options for monitoring a filesystem for changes: - -1. **Option 1**: Recursive Traversal with ReadDir - - This is quite computationally intensive but creates an accurate record of the filesystem - - Requires periodic scanning to remain updated - - Low overhead and straightforward implementation. -2. **Option 2**: Use File System Monitoring (Real-Time or Periodic Check) such as `fsnotify` - - This allows for event-based reactions to filesystem changes. - - Requires extra overhead. - - Relies on OS level features and behavior differs between OS's - - Requires OS-level configuration to ulimits in order to properly watch a large filesystem. -3. **Option 3**: Directory Metadata Heuristics. - - Using ModTime to determine when directory structures change. - - Has minimal insight into actual file changes. - - Much faster to scan for changes than Recursive transversal. - -Ultimately, FileBrowser Quantum uses a combination of 1 and 3 to perform index updates. Using something like fsnotify is a non-starter for large filesystems, where it would require manual host OS tuning to work at all. Besides, I can essentially offer the same behavior by creating "watchers" for top-level folders (a feature to come in the future). However, right now there is a single root-level watcher that works over the entire index. - -The main disadvantage of the approach is the delay caused by the scanning interval. - -### How to manually refresh the index? - -There is currently no way to manually trigger a new full indexing. This will come in a future release when the "jobs" functionality is added back. - -However, if you want to force-refresh a certain directory, this happens every time you **view it** in the UI or via the resources API. - -This also means the resources API is always up to date with the current status of the filesystem. When you "look" at a specific folder, you are causing the index to be refreshed at that location. - -### What information does the index have? - -You can see what the index looks like by using the resources API via the GET method, which returns individual directory information -- all of this information is stored in the index. - -Here is an example: - -``` -{ - "name": "filebrowser", - "size": 274467914, - "modified": "2024-11-23T19:18:57.68013727-06:00", - "type": "directory", - "files": [ - { - "name": ".dockerignore", - "size": 73, - "modified": "2024-11-20T18:14:44.91135413-06:00", - "type": "blob" - }, - { - "name": ".DS_Store", - "size": 6148, - "modified": "2024-11-22T14:45:15.901211088-06:00", - "type": "blob" - }, - { - "name": ".gitignore", - "size": 455, - "modified": "2024-11-23T19:18:57.616132373-06:00", - "type": "blob" - }, - { - "name": "CHANGELOG.md", - "size": 9325, - "modified": "2024-11-23T19:18:57.616646332-06:00", - "type": "text" - }, - { - "name": "Dockerfile", - "size": 769, - "modified": "2024-11-23T19:18:57.616941333-06:00", - "type": "blob" - }, - { - "name": "Dockerfile.playwright", - "size": 542, - "modified": "2024-11-23T19:18:57.617151875-06:00", - "type": "blob" - }, - { - "name": "makefile", - "size": 1311, - "modified": "2024-11-23T19:18:57.68017352-06:00", - "type": "blob" - }, - { - "name": "README.md", - "size": 10625, - "modified": "2024-11-23T19:18:57.617464334-06:00", - "type": "text" - } - ], - "folders": [ - { - "name": ".git", - "size": 60075460, - "modified": "2024-11-24T14:44:42.52180215-06:00", - "type": "directory" - }, - { - "name": ".github", - "size": 11584, - "modified": "2024-11-20T18:14:44.911805335-06:00", - "type": "directory" - }, - { - "name": "backend", - "size": 29247172, - "modified": "2024-11-23T19:18:57.667109624-06:00", - "type": "directory" - }, - { - "name": "docs", - "size": 14272, - "modified": "2024-11-24T13:46:12.082024018-06:00", - "type": "directory" - }, - { - "name": "frontend", - "size": 185090178, - "modified": "2024-11-24T14:44:39.880678934-06:00", - "type": "directory" - } - ], - "path": "/filebrowser" -} -``` - -### Can I disable the index and still use FileBrowser Quantum? - -You can disable the index by setting `indexing: false` in your config file. You will still be able to browse your files, but the search will not work and you may run into issues as it's not intended to be used without indexing. - -I'm not sure why you would run it like this, if you have a good reason please open an issue request on how you would like it to work -- and why you would run it without the index. diff --git a/docs/migration.md b/docs/migration.md deleted file mode 100644 index 1da2a228..00000000 --- a/docs/migration.md +++ /dev/null @@ -1,25 +0,0 @@ -# Migration help - -It is possible to use the same database as used by filebrowser/filebrowser, -but you will need to follow the following process: - -1. Create a configuration file as mentioned above. -2. Copy your database file from the original filebrowser to the path of - the new one. -3. Update the configuration file to use the database (under server in - filebrowser.yml) -4. If you are using docker, update the docker-compose file or docker run - command to use the config file as described in the install section - above. -5. If you are not using docker, just make sure you run filebrowser -c - filebrowser.yml and have a valid filebrowser config. - - -Note: share links will not work and will need to be re-created after migration. - -The filebrowser Quantum application should run with the same user and rules that -you have from the original. But keep in mind the differences that may not work -the same way, but all user configuration should be available. - -The windows binary is particularly untested, I would advise using docker if testing on windows. - diff --git a/docs/questions.md b/docs/questions.md deleted file mode 100644 index a57cd616..00000000 --- a/docs/questions.md +++ /dev/null @@ -1,50 +0,0 @@ -# Q&A topics - -## When will there be a stable release? - -Please see [roadmap](roadmap.md) for future plans -- yes a stable release will eventually come. - -## Can you add FileBrowser Quantum features to the original file browser? - -This "Quantum" version fork was created because I wanted certain features that are a dramatic and opinionated departure from the OG filebrowser. If you look at the original filebrowser repo pull requests, you will find there are many basic features that remain open for many months or years with very little attention. - -If the original filebrowser maintainers were more active and if I didn't have to worry about spending months or years playing politics about the concequences of these drastic changes, I would contribute to the original repository. - -However, **I will not make an modifications to the original filebrowser**, for these reasons: - -1. My changes are opinionated and I want full control over the experience (and consequences) of the changes. For example I removed the terminal, runners, command line flags, and more. These are changes that would probably be highly contested. I think the experience is better without them, or with the changes -- and I hope you agree. -2. Contributing to that open source project takes a long time, I may never see my changes actually make it in and don't want to waste my time trying for years, I only have so much time. -3. This project was originally a fork, but that quickly changed. There are hundereds of thousands of changes and complete departures from the original codebase. I can't simply "port" the features I write on this repo over to the OG file browser. - -Both of these repos being open source means YOU can migrate these features if you want! Feel free to spend the effort and do so for the community. I am not the only one capable of doing it and I encourage this if you have the time, energy, and knowhow to do it. - -## I notice a lot of things that don't work like the original file browser repo, how can I get this fixed? - -Please open issues and/or pull requests from a forked repo if you notice issues that should be fixed. Some changes are intentional and I may have left things broken. For example, I am not fully confident user rules work in 100% of places. When you notice things, please let me know and I will check if its an intentional change or a bug. - -## Is there a way to donate or support this project? - -Nope... not yet! It's still "unfinished" in my opinion, so I don't want to ask for any money from it. But if you have a strong desire to donate, email info@quantumx-apps.com to get in touch. - -## Is there an email or phone I can contact? - -Yes - Please contact info@quantumx-apps.com for any off-github topics. If its related to a specific application or repo such as this filebrowser, please open an issue or pull request instead. Email is only for corrospondance unrelated to technical changes or issues. - -## Can I fork this repo and use it? - -This repo has the same license as the original filebrowser, apache-2.0. Feel free to use in any way that follows the license. I have no issues with anything personally -- its open source please do as you like. However, since this is a fork of the OG repo, I am not sure what the consequences are for a fork of this repo. - -## Who are the maintainers for FileBrowser Quantum? - -Right now, just me as a personal hobby and some small contributions from the community. Once I can release a confident stable version, I plan to publicize this application more on social media. Hopefully, in the future I could pick up some extra contributors. - -I'm not looking for contributors at the moment, but if you want to me a contributor feel free to email me at info@quantumx-apps.com to see about getting contributor access. - -## Are there plans to charge for this product? - -No, this repo and project will always be free to use. - -## Is there a discord for this fork? - -Not yet, generally most interactions should happen on github for now. - diff --git a/docs/roadmap.md b/docs/roadmap.md deleted file mode 100644 index 855ce50f..00000000 --- a/docs/roadmap.md +++ /dev/null @@ -1,46 +0,0 @@ -# Planned Roadmap - -Upcoming 0.3.x releases, ordered by priority: - - Bring Themes and Branding back. - - openoffice support https://github.com/filebrowser/filebrowser/pull/2954 - - More filetype previews: eg. raw img, office, photoshop, vector, 3d files. - - Introduce jobs as replacement to runners. - - Add Job status to the sidebar - - index status. - - Job status from users - - upload status - - Opentelemetry metrics - - user access, - - file access - - download count - - last login - - more sign in support - - LDAP - - 2FA - - SSO - -Upcoming 0.4.x release: - - Support for multiple filesystem sources https://github.com/filebrowser/filebrowser/issues/2514 - - Onboarding process to add sources and configure them on first run. - - More indexing flexability - - option not to index hidden files/folders - - options folders to include/exclude from indexing - - implement more indexing runners for more efficienct filesystem watching - - tags support - -Stable release (v1.0.0) - Planned 2025: - - Once under the hood changes for things like multiple sources, jobs support, etc - - More robust backend and frontend testing - - Currently a stable release does not exist primarily because things are still changing, configuration changes are happening frequently and will for the next - - Rebrand to QuantumX App suite umbrella branding and github repo change. - -Unplanned Future releases: - - Add tools to sidebar - - duplicate file detector. - - bulk rename https://github.com/filebrowser/filebrowser/issues/2473 - - support more source types such as minio, s3, and backblaze sources https://github.com/filebrowser/filebrowser/issues/2544 - - Activity Log - - Comments support - - Trash Support - - starred/pinned files - - event based notifications diff --git a/frontend/package.json b/frontend/package.json index 988da94a..f9abca0a 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -10,6 +10,7 @@ "scripts": { "dev": "vite dev", "build": "vite build && cp -r dist/* ../backend/http/embed", + "build-windows": "vite build && robocopy dist ../backend/http/embed /e", "build-docker": "vite build", "watch": "vite build --watch", "typecheck": "vue-tsc -p ./tsconfig.json --noEmit", @@ -24,8 +25,6 @@ "clipboard": "^2.0.4", "css-vars-ponyfill": "^2.4.3", "file-loader": "^6.2.0", - "material-icons": "^1.10.5", - "material-symbols": "^0.27.2", "normalize.css": "^8.0.1", "qrcode.vue": "^3.4.1", "vue": "^3.4.21", diff --git a/frontend/playwright.config.ts b/frontend/playwright.config.ts index 44bd0f8e..9ab87e19 100644 --- a/frontend/playwright.config.ts +++ b/frontend/playwright.config.ts @@ -14,7 +14,7 @@ export default defineConfig({ /* Run tests in files in parallel */ fullyParallel: true, /* Fail the build on CI if you accidentally left test.only in the source code. */ - forbidOnly: !!process.env.CI, + forbidOnly: Boolean(process.env.CI), /* Retry on CI only */ retries: process.env.CI ? 2 : 0, /* Opt out of parallel tests on CI. */ diff --git a/frontend/public/index.html b/frontend/public/index.html index e5e5f562..c03e1b87 100644 --- a/frontend/public/index.html +++ b/frontend/public/index.html @@ -13,6 +13,8 @@ + + @@ -30,8 +32,7 @@ + + \ No newline at end of file diff --git a/frontend/src/views/bars/Default.vue b/frontend/src/views/bars/Default.vue index 055f4694..e2f333f4 100644 --- a/frontend/src/views/bars/Default.vue +++ b/frontend/src/views/bars/Default.vue @@ -20,7 +20,6 @@ import router from "@/router"; import { state, mutations, getters } from "@/store"; import { filesApi } from "@/api"; import Action from "@/components/Action.vue"; -import css from "@/utils/css"; export default { name: "listingView", @@ -134,8 +133,6 @@ export default { }, mounted() { - // Check the columns size for the first time. - this.colunmsResize(); // How much every listing item affects the window height this.setItemWeight(); @@ -198,15 +195,6 @@ export default { // How much every listing item affects the window height this.itemWeight = this.$refs.listingView.offsetHeight / itemQuantity; }, - colunmsResize() { - // Update the columns size based on the window width. - let columns = Math.floor( - document.querySelector("main").offsetWidth / this.columnWidth - ); - let items = css(["#listingView .item", "#listingView .item"]); - if (columns === 0) columns = 1; - items.style.width = `calc(${100 / columns}%)`; - }, action() { if (this.show) { mutations.showHover(this.show); diff --git a/frontend/src/views/files/ListingView.vue b/frontend/src/views/files/ListingView.vue index 1f811839..324b2609 100644 --- a/frontend/src/views/files/ListingView.vue +++ b/frontend/src/views/files/ListingView.vue @@ -1,5 +1,5 @@