Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

merge progress on writing out FileArchives #301

Merged
merged 65 commits into from
May 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
7ede2f7
initial implementation of SaveFileModel() and IStreamSaveable
nam20485 Apr 20, 2024
f941006
Merge branch 'nam20485' into write-out
nam20485 Apr 20, 2024
26f5784
Merge branch 'nam20485' into write-out
nam20485 Apr 22, 2024
56964f5
remove duplicated #include's
nam20485 Apr 23, 2024
494d750
add ISaveable interface
nam20485 Apr 23, 2024
d22717f
use static_cast instead of c-style cast
nam20485 Apr 23, 2024
4f05e6f
create ISaveable hierarchy for saving the FileArchive and its constit…
nam20485 Apr 23, 2024
de5af90
compression and moving of saved file archive working
nam20485 Apr 23, 2024
32698f6
rename to compress_dir() and use top-level directory name as the root…
nam20485 Apr 24, 2024
6758dac
hide compress_dir() in ArchiveExtractor class member
nam20485 Apr 24, 2024
a5ff4d4
call archive_write_finish_entry after writing each entry
nam20485 Apr 24, 2024
244c647
rename file variable
nam20485 Apr 25, 2024
7bc4267
implement Save(ostream) for two classes
nam20485 Apr 25, 2024
6e4a0da
implement list of directories to skip deleting during tests
nam20485 Apr 25, 2024
1cdb12c
add FileReader class test cases
nam20485 Apr 25, 2024
e974448
add and use FileReader class to hide implementation details for readi…
nam20485 Apr 25, 2024
3867524
use binary mode and unbuffered for reading files during compression
nam20485 Apr 25, 2024
09fad7d
add archive test cases
nam20485 Apr 25, 2024
ace4b68
fix FileReader test cases and actually use buffered strategy for buff…
nam20485 Apr 25, 2024
0d20724
add methods for getting test data files dir
nam20485 Apr 25, 2024
b7d88ca
use enum instead of bool for BufferStrategy
nam20485 Apr 25, 2024
64a7618
rename constant
nam20485 Apr 25, 2024
69bccbb
use enum for buffer strategy
nam20485 Apr 25, 2024
e9ea965
cleanup
nam20485 Apr 25, 2024
6a4604e
fix error
nam20485 Apr 25, 2024
2548a47
add EnumMap class
nam20485 Apr 25, 2024
7ba31b9
add contains() methods to EnumMap
nam20485 Apr 26, 2024
296e009
minor refactor
nam20485 Apr 26, 2024
7b09ea7
add str_icmp()
nam20485 Apr 26, 2024
bc8df1c
export RgbColor type
nam20485 Apr 26, 2024
dd25a8a
matrix file Save() complete- handles enums
nam20485 Apr 26, 2024
dbf268c
remove case-insensitive compare kruft from EnumMap
nam20485 Apr 26, 2024
24247b4
add IsLinux() andf IsApple() functions
nam20485 Apr 26, 2024
a347c84
add empty CrossPlatform class
nam20485 Apr 26, 2024
deb1f74
put EnumMap class into Utils namespace
nam20485 Apr 26, 2024
3ab7c53
add failed attempt at find_str_icmp()
nam20485 Apr 26, 2024
893f412
add macro defines for IS_WINDOWS, etc.
nam20485 Apr 26, 2024
092e62a
implement localtime_safe
nam20485 Apr 26, 2024
58350cd
add transitive header include excplicitly
nam20485 Apr 26, 2024
0747542
return const& bc we don't need to make a copy of the string values we…
nam20485 Apr 27, 2024
a3cdf0b
minor refactor
nam20485 Apr 27, 2024
96cd0d2
add find_str_icmp() methods
nam20485 Apr 27, 2024
f924776
add #include header
nam20485 Apr 27, 2024
f4ca2c2
remove unused header include
nam20485 Apr 27, 2024
daf86a7
Merge branch 'nam20485' into write-out
nam20485 Apr 27, 2024
f967dc5
fix build command
nam20485 Apr 27, 2024
0f6f791
fix some warnings/errors on linux
nam20485 Apr 27, 2024
ea1719e
add windows implementations of getenv_safe() and tmpname_safe()
nam20485 Apr 27, 2024
2e51ce3
remove commented code
nam20485 Apr 27, 2024
0e0b0b9
remove commented code
nam20485 Apr 27, 2024
feb74e3
Merge branch 'write-out' of github.com:nam20485/OdbDesign into write-out
nam20485 Apr 27, 2024
47a6736
rename fastcopy and copy to fastmove and move since they remove the f…
nam20485 Apr 27, 2024
7d75ff3
use crow::status enum values instead of "magic" number literals
nam20485 Apr 28, 2024
352ae4f
refactor fast_move() to use error_code's instead of relying on except…
nam20485 Apr 28, 2024
9219244
fix bug where struct stat instance was being used prior to being init…
nam20485 Apr 28, 2024
387b049
add #includes
nam20485 Apr 28, 2024
418f864
add 'pass-through' implementations for Linux and Apple platforms that…
nam20485 Apr 28, 2024
de9c466
add test cases for CrossPlatform methods
nam20485 Apr 28, 2024
60f79da
add test case for ArchiveExtractor::CompressDir method
nam20485 Apr 28, 2024
2bfa68a
refactor test data-specific functionality out of FileArchiveLoadFixtu…
nam20485 Apr 28, 2024
f99fb69
use actual buffered reads when reading files to compress
nam20485 Apr 28, 2024
1989393
remove comments
nam20485 Apr 29, 2024
9fa360c
add #include headers
nam20485 Apr 29, 2024
ff08354
add stepname dircetory under steps dir
nam20485 Apr 30, 2024
96498af
implement Save() in NetlistFile
nam20485 Apr 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions OdbDesignLib/App/BasicRequestAuthentication.cpp
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#include "BasicRequestAuthentication.h"
#include <string>
#include "macros.h"
#include <cstdlib>

using namespace Utils;

Expand All @@ -17,21 +18,20 @@ namespace Odb::Lib::App
if (resp.code != crow::status::OK)
{
const auto& authHeader = req.get_header_value(AUTHORIZATION_HEADER_NAME);
if (authHeader.empty()) return crow::response(401, "Unauthorized");
if (authHeader.empty()) return crow::response(crow::status::UNAUTHORIZED, "Unauthorized");

auto authValue = authHeader.substr(6);
if (authValue.empty()) return crow::response(401, "Unauthorized");
if (authValue.empty()) return crow::response(crow::status::UNAUTHORIZED, "Unauthorized");

auto authValueDecoded = crow::utility::base64decode(authValue, authValue.size());
if (authValueDecoded.empty()) return crow::response(401, "Unauthorized");
if (authValueDecoded.empty()) return crow::response(crow::status::UNAUTHORIZED, "Unauthorized");

auto seperatorPos = authValueDecoded.find(':');
if (seperatorPos == std::string::npos) return crow::response(401, "Unauthorized");
if (seperatorPos == std::string::npos) return crow::response(crow::status::UNAUTHORIZED, "Unauthorized");

auto username = authValueDecoded.substr(0, seperatorPos);
auto password = authValueDecoded.substr(seperatorPos + 1);

//if (! VerifyCredentials(username, password)) return crow::response(403, "Invalid username or password");
resp = VerifyCredentials(username, password);
}
return resp;
Expand All @@ -58,10 +58,10 @@ namespace Odb::Lib::App
if (username != validUsername ||
password != validPassword)
{
return crow::response(403, "Invalid username or password");
return crow::response(crow::status::FORBIDDEN, "Invalid username or password");
}

// 200 Authorized!
return crow::response(200, "Authorized");
return crow::response(crow::status::OK, "Authorized");
}
}
1 change: 1 addition & 0 deletions OdbDesignLib/App/DesignCache.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
#include <exception>
#include <filesystem>
#include <utility>
#include <StringVector.h>

using namespace Utils;
using namespace std::filesystem;
Expand Down
20 changes: 16 additions & 4 deletions OdbDesignLib/App/OdbAppBase.cpp
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
#include "OdbServerAppBase.h"
#include "Logger.h"
#include <filesystem>

using namespace Utils;
using namespace std::filesystem;

namespace Odb::Lib::App
{
Expand Down Expand Up @@ -46,17 +48,27 @@ namespace Odb::Lib::App
try
{
auto pFileArchive =
designs().GetDesign(args().loadDesign());
//designs().GetFileArchive(args().loadDesign());
//designs().GetDesign(args().loadDesign());
designs().GetFileArchive(args().loadDesign());
if (pFileArchive == nullptr)
{
logerror("Failed to load design specified in arguments \"" + args().loadDesign() + "\"");
return Utils::ExitCode::FailedInitLoadDesign;
}
else
{
//std::string filename =
pFileArchive->SaveFileModel(".", "notused");
}
}
catch (filesystem_error& fe)
{
logexception(fe);
logerror("filesystem_error: \"" + args().loadDesign() + "\" " + fe.what());
}
catch (std::exception&)
catch (std::exception& e)
{
//logexception(e);
logexception(e);
logerror("Failed to load design specified in arguments \"" + args().loadDesign() + "\"");
return Utils::ExitCode::FailedInitLoadDesign;
}
Expand Down
6 changes: 3 additions & 3 deletions OdbDesignLib/App/RequestAuthenticationBase.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -16,16 +16,16 @@ namespace Odb::Lib::App
if (IsDebug() && IsLocal())
{
// 200 Authorized!
return crow::response(200, "Authorized");
return crow::response(crow::status::OK, "Authorized");
}
else if (m_disableAuthentication)
{
// 200 Authorized!
return crow::response(200, "Authorized");
return crow::response(crow::status::OK, "Authorized");
}
else
{
return crow::response(401, "Unauthorized");
return crow::response(crow::status::UNAUTHORIZED, "Unauthorized");
}
}
}
2 changes: 1 addition & 1 deletion OdbDesignLib/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ add_library(OdbDesign SHARED
"ProtoBuf/via.pb.h" "ProtoBuf/via.pb.cc"
"ProtoBuf/package.pb.h" "ProtoBuf/package.pb.cc"
"FileModel/parse_error.h" "FileModel/parse_info.h" "FileModel/parse_info.cpp" "FileModel/parse_error.cpp" "FileModel/invalid_odb_error.h" "FileModel/invalid_odb_error.cpp" "ProtoBuf/common.pb.h" "ProtoBuf/common.pb.cc" "ProtoBuf/componentsfile.pb.h" "ProtoBuf/componentsfile.pb.cc" "FileModel/Design/PropertyRecord.h" "FileModel/Design/PropertyRecord.cpp" "FileModel/Design/FeaturesFile.h" "FileModel/Design/FeaturesFile.cpp" "FileModel/Design/ContourPolygon.h" "FileModel/Design/ContourPolygon.cpp" "FileModel/Design/SymbolName.h" "FileModel/Design/SymbolName.cpp" "FileModel/Design/SymbolsDirectory.h" "FileModel/Design/SymbolsDirectory.cpp" "FileModel/Design/StepHdrFile.h" "FileModel/Design/StepHdrFile.cpp" "FileModel/Design/AttributeLookupTable.h" "FileModel/Design/AttributeLookupTable.cpp"
"App/RequestAuthenticationBase.h" "App/RequestAuthenticationBase.cpp" "App/BasicRequestAuthentication.h" "App/BasicRequestAuthentication.cpp")
"App/RequestAuthenticationBase.h" "App/RequestAuthenticationBase.cpp" "App/BasicRequestAuthentication.h" "App/BasicRequestAuthentication.cpp" "FileModel/IStreamSaveable.h")

# disable warning C4250: inheritance by dominance
target_compile_options(OdbDesign PUBLIC
Expand Down
24 changes: 16 additions & 8 deletions OdbDesignLib/FileModel/Design/AttrListFile.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -43,20 +43,20 @@ namespace Odb::Lib::FileModel::Design

loginfo("checking for extraction...");

std::filesystem::path featuresFilePath;
for (const std::string featuresFilename : ATTRLIST_FILENAMES)
std::filesystem::path attrListFilePath;
for (const std::string attrListFilename : ATTRLIST_FILENAMES)
{
loginfo("trying attrlist file: [" + featuresFilename + "]...");
loginfo("trying attrlist file: [" + attrListFilename + "]...");

featuresFilePath = Utils::ArchiveExtractor::getUncompressedFilePath(m_directory, featuresFilename);
if (exists(featuresFilePath) && is_regular_file(featuresFilePath))
attrListFilePath = Utils::ArchiveExtractor::getUncompressedFilePath(m_directory, attrListFilename);
if (exists(attrListFilePath) && is_regular_file(attrListFilePath))
{
loginfo("found attrlist file: [" + featuresFilePath.string() + "]");
loginfo("found attrlist file: [" + attrListFilePath.string() + "]");
break;
}
}

m_path = featuresFilePath;
m_path = attrListFilePath;

loginfo("any extraction complete, parsing data...");

Expand Down Expand Up @@ -203,6 +203,14 @@ namespace Odb::Lib::FileModel::Design
return true;
}


bool AttrListFile::Save(std::ostream& os)
{
os << Constants::UNITS_TOKEN << " = " << m_units << std::endl;
for (const auto& kvAttribute : m_attributesByName)
{
os << kvAttribute.first << " = " << kvAttribute.second << std::endl;
}

return true;
}
} // namespace Odb::Lib::FileModel::Design
6 changes: 4 additions & 2 deletions OdbDesignLib/FileModel/Design/AttrListFile.h
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,11 @@
#include "../../IProtoBuffable.h"
#include "../../ProtoBuf/attrlistfile.pb.h"
#include "../../odbdesign_export.h"
#include "../IStreamSaveable.h"

namespace Odb::Lib::FileModel::Design
{
class ODBDESIGN_EXPORT AttrListFile : public IProtoBuffable<Odb::Lib::Protobuf::AttrListFile>
class ODBDESIGN_EXPORT AttrListFile : public IProtoBuffable<Odb::Lib::Protobuf::AttrListFile>, public IStreamSaveable
{
public:
AttrListFile();
Expand All @@ -24,6 +25,8 @@ namespace Odb::Lib::FileModel::Design
const AttributeMap& GetAttributes() const;

bool Parse(std::filesystem::path directory);
// Inherited via IStreamSaveable
bool Save(std::ostream& os) override;

// Inherited via IProtoBuffable
std::unique_ptr<Odb::Lib::Protobuf::AttrListFile> to_protobuf() const override;
Expand All @@ -40,7 +43,6 @@ namespace Odb::Lib::FileModel::Design
inline static const auto ATTRLIST_FILENAMES = { "attrlist" };
inline static const char* OPTIONAL_ATTRIBUTES[] = { "" };
};

}

#endif //ODBDESIGN_ATTRLISTFILE_H
11 changes: 5 additions & 6 deletions OdbDesignLib/FileModel/Design/ComponentsFile.cpp
Original file line number Diff line number Diff line change
@@ -1,10 +1,4 @@
#include "ComponentsFile.h"
#include "ComponentsFile.h"
#include "ComponentsFile.h"
#include "ComponentsFile.h"
#include "ComponentsFile.h"
#include "ComponentsFile.h"
#include "ComponentsFile.h"
#include <fstream>
#include <sstream>
#include <filesystem>
Expand Down Expand Up @@ -155,6 +149,11 @@ namespace Odb::Lib::FileModel::Design
}
}

bool ComponentsFile::Save(std::ostream& os)
{
return true;
}

std::unique_ptr<Odb::Lib::Protobuf::ComponentsFile::BomDescriptionRecord> ComponentsFile::BomDescriptionRecord::to_protobuf() const
{
std::unique_ptr<Odb::Lib::Protobuf::ComponentsFile::BomDescriptionRecord> pBomDescriptionRecordMessage(new Odb::Lib::Protobuf::ComponentsFile::BomDescriptionRecord);
Expand Down
18 changes: 11 additions & 7 deletions OdbDesignLib/FileModel/Design/ComponentsFile.h
Original file line number Diff line number Diff line change
Expand Up @@ -11,17 +11,20 @@
#include "PropertyRecord.h"
#include "../../ProtoBuf/componentsfile.pb.h"
#include "AttributeLookupTable.h"
#include "../IStreamSaveable.h"


namespace Odb::Lib::FileModel::Design
{
class ODBDESIGN_EXPORT ComponentsFile : public IProtoBuffable<Odb::Lib::Protobuf::ComponentsFile>
class ODBDESIGN_EXPORT ComponentsFile : public IProtoBuffable<Odb::Lib::Protobuf::ComponentsFile>, public IStreamSaveable
{
public:
ComponentsFile();
~ComponentsFile();

bool Parse(std::filesystem::path directory);
// Inherited via IStreamSaveable
bool Save(std::ostream& os) override;

std::string GetUnits() const;
BoardSide GetSide() const;
Expand Down Expand Up @@ -123,7 +126,7 @@ namespace Odb::Lib::FileModel::Design

constexpr inline static const char* TOP_COMPONENTS_LAYER_NAME = "comp_+_top";
constexpr inline static const char* BOTTOM_COMPONENTS_LAYER_NAME = "comp_+_bot";

// Inherited via IProtoBuffable
std::unique_ptr<Odb::Lib::Protobuf::ComponentsFile> to_protobuf() const override;
void from_protobuf(const Odb::Lib::Protobuf::ComponentsFile& message) override;
Expand All @@ -148,9 +151,9 @@ namespace Odb::Lib::FileModel::Design
const bool m_allowToepintNetNumbersOfNegative1 = true;

constexpr inline static const char* COMPONENTS_FILENAMES[] =
{
"components",
"components2",
{
"components",
"components2",
"components3"
};

Expand All @@ -168,6 +171,7 @@ namespace Odb::Lib::FileModel::Design
constexpr inline static const char* BOM_DESCR_RECORD_TOKEN_VPL_VND = "VPL_VND";
constexpr inline static const char* BOM_DESCR_RECORD_TOKEN_VPL_MPN = "VPL_MPN";
constexpr inline static const char* BOM_DESCR_RECORD_TOKEN_VND = "VND";
constexpr inline static const char* BOM_DESCR_RECORD_TOKEN_MPN = "MPN";
};
constexpr inline static const char* BOM_DESCR_RECORD_TOKEN_MPN = "MPN";

};
}
7 changes: 6 additions & 1 deletion OdbDesignLib/FileModel/Design/EdaDataFile.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -366,7 +366,12 @@ namespace Odb::Lib::FileModel::Design
pFeatureGroupRecord->from_protobuf(featureGroupRecordMessage);
m_featureGroupRecords.push_back(pFeatureGroupRecord);
}
}
}

bool EdaDataFile::Save(std::ostream& os)
{
return true;
}

bool EdaDataFile::Parse(std::filesystem::path path)
{
Expand Down
7 changes: 5 additions & 2 deletions OdbDesignLib/FileModel/Design/EdaDataFile.h
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,12 @@
#include "PropertyRecord.h"
#include "ContourPolygon.h"
#include "AttributeLookupTable.h"
#include "../IStreamSaveable.h"


namespace Odb::Lib::FileModel::Design
{
class ODBDESIGN_EXPORT EdaDataFile : public IProtoBuffable<Odb::Lib::Protobuf::EdaDataFile>
class ODBDESIGN_EXPORT EdaDataFile : public IProtoBuffable<Odb::Lib::Protobuf::EdaDataFile>, public IStreamSaveable
{
public:
EdaDataFile(bool logAllLineParsing = false);
Expand All @@ -27,7 +28,9 @@ namespace Odb::Lib::FileModel::Design
const std::string& GetUnits() const;
const std::string& GetSource() const;

bool Parse(std::filesystem::path path);
bool Parse(std::filesystem::path path);
// Inherited via IStreamSaveable
bool Save(std::ostream& os) override;

struct ODBDESIGN_EXPORT FeatureIdRecord : public IProtoBuffable<Odb::Lib::Protobuf::EdaDataFile::FeatureIdRecord>
{
Expand Down
6 changes: 5 additions & 1 deletion OdbDesignLib/FileModel/Design/FeaturesFile.cpp
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
#include "FeaturesFile.h"
#include "FeaturesFile.h"
#include "ArchiveExtractor.h"
#include <fstream>
#include "Logger.h"
Expand Down Expand Up @@ -858,6 +857,11 @@ namespace Odb::Lib::FileModel::Design
}
}

bool FeaturesFile::Save(std::ostream& os)
{
return true;
}

FeaturesFile::FeatureRecord::~FeatureRecord()
{
m_contourPolygons.clear();
Expand Down
5 changes: 4 additions & 1 deletion OdbDesignLib/FileModel/Design/FeaturesFile.h
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,12 @@
#include "../../ProtoBuf/featuresfile.pb.h"
#include "SymbolName.h"
#include "AttributeLookupTable.h"
#include "../IStreamSaveable.h"


namespace Odb::Lib::FileModel::Design
{
class FeaturesFile : public IProtoBuffable<Odb::Lib::Protobuf::FeaturesFile>
class FeaturesFile : public IProtoBuffable<Odb::Lib::Protobuf::FeaturesFile>, public IStreamSaveable
{
public:
FeaturesFile();
Expand Down Expand Up @@ -86,6 +87,8 @@ namespace Odb::Lib::FileModel::Design
};

bool Parse(std::filesystem::path directory, const std::string& alternateFilename = "");
// Inherited via IStreamSaveable
bool Save(std::ostream& os) override;

std::string GetUnits() const;
std::filesystem::path GetPath();
Expand Down
Loading