Ismail Seyfi's Blog

There are many way to query JSON data. This article focuses on one of those possible ways using few developer utility tools.  jqplay.org :  https://jqplay.org/s/nwxNZ7LOY6 sqlite3:  https://www.sqlite.org/cli.html sqlite-utils:  https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-json-data You can call an HTTP API as below. Use jq to transform the return content as you need. Then pipe it through sqlite-utils to perform upsert. Following code snippet simply creates a "test" table with primary key being the id attribute.  curl --location --request POST 'https://seyfi.net/searches?size=1000&page=0' \ --header 'Authorization: Bearer XXXXXX' \ --header 'Content-Type: application/json' \ --data-raw '{"date":"2019-01-01T00:00:00.000Z"}' | jq '.content' | sqlite-utils upsert /tmp/test.db content - --pk=id Once this executes, you can then filter this and export easily using Sqlite3 as follows. >sqlite3...

In an earlier post, I described my attempt to save some generated excel documents to OneDrive using OAUTH 2.0 client_credentials through the Graph API. That blog post is located here . OneDrive is for individual use. In our organization, we also use SharePoint. SharePoint 365 also uses OneDrive for its backend file system. However, Microsoft has different APIs to save data into SharePoint sites through its Office 365 APIs. This blog post will explain how you can configure your Azure Portal and Office 365 to save documents to SharePoint programatically. Overview of Technical Approach Office 365/SharePoint provide OAUTH 2.0 enabled APIs for access. In order to use such APIs, applications need to be registered with our Azure Management Portal. Once applications are registered, OAUTH 2.0 with client certificates will be used to eliminate user consent flow as we will be using these APIs programmatically by back-end applications. There will be no end-user logging into SharePoint. Here ...

I recently had the opportunity to use Office 365 / OneDrive. I was trying to programmatically save some generated (excel) documents to a folder in SharePoint / OneDrive using a back-end Java utility. Hopefully, this write up captures all of the details of how I was able to accomplish it.  Overview of Technical Approach Office 365 and OneDrive provide OAUTH 2.0 enabled APIs for access. In order to use such APIs, applications need to be registered with Azure Management Portal. Once applications are registered, OAUTH 2.0 grant_type of client_credentials can be used to eliminate user consent flow as you can use these APIs programmatically by back-end applications.  There will be no end-user logging into OneDrive. Here is the documentation provided by OneDrive that talks about how to create and register your application in order to use these APIS.  https://dev.onedrive.com/app-registration.htm#register-your-app-for-onedrive-for-business Once an application is regist...

Remote debugging capabilities of IDEs can come in handy when you are unable to replicate certain situations locally. If your only option is to debug applications that are running in remote servers, you will have to make sure you have met few requirements: Remote applications are started in debug mode You know these applications' debug ports so you can make connections to them Network is configured to allow you to connect to these server through the identified ports Recently I found myself in a situation like this where I needed to have my local environment connect to a machine that was running in AWS EC2. This application happens to be WSO2 Identity Server which was deployed into WSO2's Carbon Server. The first thing I had to do was to start WSO2 in debug mode. I was able to do this by running the following command: IS_HOME/bin/wso2server.sh -debug 8005 (you can specify a different port here if you like) Once the above command is executed, WSO2 will be ready to ...

I just completed by Big Data Processing Using Hadoop class at John's Hopkins. The class started with a general overview of the Apache Hadoop platform with HDFS (Hadoop File System), Map Reduce framework. We dived into Pig, Hive and few other components. We completed 3 decent size projects and numerous lab assignments. Lab assignments were designed to get us introduced to these topics while assignments were designed to give us real life scenario problems and how this very complicated ecosystem of tools and ideas can be used to solve these problems. I very much enjoyed the topics and assignments in this class. We also had an opportunity to choose our own topic (outside of what was being taught in class) and present it in class. I thought this was a really nice opportunity to dive into one of those other Apache projects that compliment the Hadoop platform. Working on few of the assignments, it become clear to me that we really need a way to organize Hadoop Jobs into a workflow and ...