From 4ca24351592f939bda64d28cb51cef2bba25fd47 Mon Sep 17 00:00:00 2001 From: mohamed yahia Date: Wed, 15 Nov 2023 21:01:20 +0200 Subject: [PATCH 1/3] Fixes #43 ( missing type declaration ) --- tsconfig.json | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/tsconfig.json b/tsconfig.json index 04b2fbc..98d0126 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -2,10 +2,11 @@ "compilerOptions": { "allowJs": true, "moduleResolution": "node", - "esModuleInterop": true + "esModuleInterop": true, + "declaration": true, + "composite": true }, - "files": [], - "include": [], + "include": ["src/**/*.ts"], "references": [ { "path": "./tsconfig.lib.json" From 20c4ae435197973ed0a7512e2a608c91fe5363b8 Mon Sep 17 00:00:00 2001 From: Rufus Pollock Date: Thu, 16 Nov 2023 10:13:20 +0100 Subject: [PATCH 2/3] [README,#50][s]: add feature checklist in README. summarized version of job stories in https://datahub.io/notes/markdowndb. Also added to roadmap epic. --- README.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/README.md b/README.md index 1076527..43ea9b1 100644 --- a/README.md +++ b/README.md @@ -9,6 +9,29 @@ MarkdownDB is a javascript library for treating markdown files as a database -- **🚧 MarkdownDB is in its early development stage** +## Features and Roadmap + +- [x] **Index a folder of files** - create a db index given a folder of markdown and other files + - [ ] BONUS Index multiple folders (with support for configuring e.g. prefixing in some way e.g. i have all my blog files in this separate folder over here) + - [x] **Command line tool for indexing**: Create a markdowndb (index) on the command line + +Extract structured data like: + +- [x] **Frontmatter metadata**: Extract markdown frontmatter and add in a metadata field + - [ ] deal with nested frontmatter (? what does this mean?) + - [ ] deal with casting types e.g. string, number so that we can query in useful ways e.g. find me all blog posts before date X +- [ ] **Tags**: Extracts tags in markdown pages + - [x] Extract tags in frontmatter + - [ ] Extract tags in body like `#abc` +- [ ] **Links**: links between files like `[hello](abc.md)` or wikilink style `[[xyz]]` so we can compute backlinks or deadlinks etc (see #4) +- [ ] **Tasks**: extract tasks like this `- [ ] this is a task` (See obsidian data view) + +Data enhancement and validation + +- [ ] **Computed fields**: add new metadata properties based on existing metadata e.g. a slug field computed from title field; or, adding a title based on the first h1 heading in a doc; or, a type field based on the folder of the file (e.g. these are blog posts). cf https://www.contentlayer.dev/docs/reference/source-files/define-document-type#computedfields. #54 +- [ ] **Data validation and Document Types**: validate metadata against a schema/type so that I know the data in the database is "valid" #55 + - [ ] BYOT (bring your own types): i want to create my own types ... so that when i get an object out it is cast to the right typescript type + ## Quick start ### You have a folder of markdown content From 7a261f9619ba1a62ffc9e1c99bf1c84ca30cdc99 Mon Sep 17 00:00:00 2001 From: Rufus Pollock Date: Fri, 17 Nov 2023 22:07:45 +0100 Subject: [PATCH 3/3] [README][s]: add architecture section with mermaid diagram of flow. --- README.md | 26 ++++++++++++-------------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 43ea9b1..146693c 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,8 @@ MarkdownDB is a javascript library for treating markdown files as a database -- - [x] **Index a folder of files** - create a db index given a folder of markdown and other files - [ ] BONUS Index multiple folders (with support for configuring e.g. prefixing in some way e.g. i have all my blog files in this separate folder over here) - [x] **Command line tool for indexing**: Create a markdowndb (index) on the command line + - [ ] SQL(ite) index + - [ ] JSON index Extract structured data like: @@ -276,19 +278,15 @@ mddb.getTags(); mddb.getLinks({ fileId: "ID", direction: "forward" }); ``` -## Features +## Architecture -- indexing markdown files into an SQLite database, with: - - frontmatter data extraction (specifically tags and document types) - - wiki links extraction (CommonMark links, Obsidian links and embeds) -- querying database for: - - a list of all or some of the markdown files in your content folder: e.g. get all your blog posts tagged with "tutorial" - - get backlinks to or forward on a given page, and e.g. list them in the bottom of your pages, so that users can quickly find related content +```mermaid +graph TD -## Upcoming features - -- custom document types and schemas with data validation -- computed fields -- indexing multiple folders -- extracting tasks -- and much more... +markdown --remark-parse--> st[syntax tree] +st --extract features--> jsobj1[TS Object eg. File plus Metadata plus Tags plus Links] +jsobj1 --computing--> jsobj[TS Objects] +jsobj --convert to sql--> sqlite[SQLite markdown.db] +jsobj --write to disk--> json[JSON on disk in .markdowndb folder] +jsobj --tests--> testoutput[Test results] +```