From a security perspective, to which location should one copy an app inside a docker image?

I don't see a point.

An external attacker (who has access to the host) can easily find where the files are - he only has to look at the last file system layer and will have them presented right there for him.

An internal attacker (who gains access to the container through a misconfigured part of the app or whatever) will also rather easily find them by looking through the process table at /proc and noting the CWDs.

Attacking the image might be easier with a fixed location, as the attacker only has to add another FS layer, overwriting that directory with his own application. But then, if you did not secure your image store, it does not matter anyways what else you do.

So... I'd say the convenience of a common location is good, and little to no security would be gained; and if, then only security by obscurity, which does not count.


Looking for ways to host application in AWS and have it monitored using opensource

You can try built-in tools that provide by AWS.
AWS calls them Management Tools.

Amazon Inspector

Automated security assessment service to help improve the security and compliance of applications deployed on AWS.
Amazon Inspector is an automated security assessment service that helps improve the security and compliance of applications deployed on AWS. Amazon Inspector automatically assesses applications for vulnerabilities or deviations from best practices. After performing an assessment, Amazon Inspector produces a detailed list of security findings prioritized by level of severity. These findings can be reviewed directly or as part of detailed assessment reports which are available via the Amazon Inspector console or API.
— AWS Inspector

Amazon Web Services (AWS) Monitoring With Nagios

Implementing effective Amazon Web Services monitoring with Nagios
offers the following benefits:
Increased server, services, and application availability Fast
detection of network outages Fast detection of cloud computing
environment problems
— Nagios

Some other tools:

Scalr

Management Tools by AWS


Ansible No such file or directory error message for any executable in Amazon Linux AMI

After struggling for a time I solved the problem by changing the image in the ci from Ubuntu to ansible-runner.


How to authenticate, with azure devops services, using SAML via on premise standalone java application?

There is a fairly sizable range of authentication options, based upon what you have said you are looking for a non-interactive login which rules out some options, and leaves the following:

Device Profile - probably not ideal for you given you would need to find a way to this in Java as the examples are few and far between and mostly written in C#.
TFS Application - again not ideal given you would need to either find someone who has written a 3rd party client library for Java or write one yourself.
Azure DevOps Extension - while you could write an Azure DevOps extension, and use that to access Azure DevOps it's a lot of work and potentially would expose security issues.

Given that we have probably ruled out non-interactive login to Azure DevOps you could also explore one of the two following options:

Interactive Logins - is there an option in your environment from forcing the user to authenticate with Azure DevOps using OAuth then capture the token for non-interactive use?  This is quite a common pattern with GitHub, GitLab and Bitbucket.
Personal Access Tokens - mostly this would be abusing a feature of Azure DevOps as these are designed for users not applications, although it could work in your constrained environment.
Non-interactive Login using your Identity Provider - you haven't detailed who your Identity Provider is, but it is possible it supports authentication using Client Certificates, Shared Secrets, etc. which would give you a SAML token which could be exchanged for an Azure DevOps Access token.

Hope those help.


Access S3 files from Datacenter over VPN to VPC

You need to use...

the Internet, from your data center, to access S3 directly (not via the endpoint) or
an AWS DirectConnect connection to access S3 directly (not via the endpoint) or
an HTTP proxy server inside this VPC to access S3 (via the endpoint)

...because S3 endpoints are not designed to be used across VPN connections.  They are only accessible from instances in the VPC where they exist.

  Endpoint connections cannot be extended out of a VPC. Resources on the other side of a VPN connection, VPC peering connection, AWS Direct Connect connection, or ClassicLink connection in your VPC cannot use the endpoint to communicate with resources in the endpoint service.

  https://docs.aws.amazon.com/vpc/latest/userguide/vpce-gateway.html#vpc-endpoints-limitations


How to skip rest of hosts when item found in host - Ansible

Let's test the use-case with 10 remote hosts (test_01 - test_10) in the group test and user my-user in the hosts test_03 and test_07. To test the existence of the user, use the shell command
shell> cat /etc/passwd|grep my-user

For example the playbook
shell> cat playbook.yml
- hosts: test
  tasks:
    - shell: cat /etc/passwd|grep my-user
      register: result
      ignore_errors: true
    - set_fact:
        results: "{{ results|default({})|
                     combine({item: hostvars[item].result.rc}) }}"
      loop: "{{ ansible_play_hosts }}"
      run_once: true
    - debug:
        var: results
      run_once: true

gives
ok: [test_01] =>
  results:
    test_01: 1
    test_02: 1
    test_03: 0
    test_04: 1
    test_05: 1
    test_06: 1
    test_07: 0
    test_08: 1
    test_09: 1
    test_10: 1

To skip the rest of the hosts, when the first host is found, iterate the list of the hosts and test rc. Then add the host to a new group and run the next play with this group. For example,
shell> cat playbook.yml
- hosts: test
  vars:
    dresult:
      rc: 1
  tasks:
    - block:
        - shell: cat /etc/passwd|grep my-user
          register: result
          loop: "{{ ansible_play_hosts }}"
          delegate_to: "{{ item }}"
          ignore_errors: true
          when: (result|default(dresult)).rc == 1
        - set_fact:
            my_host: "{{ result.results|json_query('[?rc == `0`].item')|first }}"
        - debug:
            var: my_host
        - add_host:
            name: "{{ my_host }}"
            groups: my_group
      run_once: true

- hosts: my_group
  tasks:
    - debug:
        var: inventory_hostname

gives
shell> ansible-playbook playbook.yml

PLAY [test] ****

TASK [shell] ****
failed: [test_01 -> test_01] (item=test_01) => changed=true
  ansible_loop_var: item
  cmd: cat /etc/passwd|grep my-user
  delta: '0:00:00.023271'
  end: '2020-09-25 08:20:31.823168'
  item: test_01
  msg: non-zero return code
  rc: 1
  start: '2020-09-25 08:20:31.799897'
  stderr: ''
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>
failed: [test_01 -> test_02] (item=test_02) => changed=true
  ansible_loop_var: item
  cmd: cat /etc/passwd|grep my-user
  delta: '0:00:00.017642'
  end: '2020-09-25 08:20:33.386657'
  item: test_02
  msg: non-zero return code
  rc: 1
  start: '2020-09-25 08:20:33.369015'
  stderr: ''
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>
changed: [test_01 -> test_03] => (item=test_03)
skipping: [test_01] => (item=test_04)
skipping: [test_01] => (item=test_05)
skipping: [test_01] => (item=test_06)
skipping: [test_01] => (item=test_07)
skipping: [test_01] => (item=test_08)
skipping: [test_01] => (item=test_09)
skipping: [test_01] => (item=test_10)
...ignoring

TASK [set_fact] ****
ok: [test_01]

TASK [debug] ****
ok: [test_01] =>
  my_host: test_03

TASK [add_host] ****
changed: [test_01]

PLAY [my_group] ****

TASK [debug] ****
ok: [test_03] =>
  inventory_hostname: test_03

PLAY RECAP *****
test_01: ok=4 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=1
test_03: ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0


Creating ec2 instances and installing mongoDB using automation

It's possible. I would use Terraform for creating the infrastructure on AWS and Ansible for provisioning. You will have to write .tf files and ansible-playbooks in order to do this.

Just to let you know, if all 3 nodes are equal you should do the installation and provisioning once and then create an AMI to set up the rest. Maybe an autoscaling group is a possibility too.

Keep in mind that AWS provides documentdb as document-oriented database as a service. It's not mongoDB but it's pretty similar (there are some functions mongo provides and docdb doesn't).


Can I run multiple pipelines in a single GitLab repo using GitLab CI/CD?

Yes, you can use the rules syntax. You can use this in combination with regex for commit message, ci_pipeline_source or any other available CI variables.

job1:
  script:
    - do something on schedule only
  rules:
    - if: '"$CI_PIPELINE_SOURCE" == "schedule"'
      when: always

job2:
  script:
    - runs on Merge request pipeline
  rules:
    - if: $CI_MERGE_REQUEST_ID
      when: always
    when: never

job2:
  script:
    - runs on changes to anything in src directory or Dockerfile
  rules:
    - changes:
        - src/*
        - Dockerfile

Note you can also include multiple CI templates in your gitlab-ci.yml. Additonally, you can include a CI template with "hidden" job templates that you can reference with extends:
# Include ci files
include:
 - local: '/templates/.scheduled-job-template.yml' # CI template with hidden job (.scheduled_job) that runs on schedule
 - local: '/templates/.lint-and-validate.yml' # contains jobs for linting and validating project

job1:
  extends:
    - .scheduled_job
  rules:
    - if: '"$CI_PIPELINE_SOURCE" == "schedule"'
      when: always

job2:
  script:
    - runs on Merge request pipeline
  rules:
    - if: $CI_MERGE_REQUEST_ID
      when: always
    when: never

In doing this you can compose the jobs/pipelines you want in its own yml file and then define the jobs using those templates in the gitlab-ci.yml, which will help keep things maintainable and clear if you are running numerous different pipeline/pipeline configurations from the same project.


What is the optimal speed relation of CI/CD jobs to a (reference) local machine?

You want the execution speed of the CI pipeline as fast as (practically) possible, irrespective of being a local or shared setup or the time required to set it up - this is what effectively dictates the overall development speed in the respective project.

But it should be noted that by speed here I don't mean the duration of each CI execution. The duration of such execution may be directly related to the coverage of the CI QA verification, which translates into stabiliity of the development process and the quality of the product. For example including a smoketest in the CI verification will increase the duration of the CI execution, but will also raise the product quality - it's a good thing.

But I have to mention: such private CI setups (even those SLA-backed CI setups you mention) are often an indication of what the ThoughtWorks team called the CI Theatre. Simply using CI tools doesn't make the process a CI one. Using project/feature/integration branches which still require subsequent branch merges (other than cherry-picked fixes) before release means development is still done in silos (those branches are the silos) - waterfall style.

Side note: Frankly, if I'd be part of a team in which management doesn't understand the importance of CI (let alone not being able to effectively communicate on such critical subject) I'd look for another job - I doubt they'll remain in business for very long ;)


Docker prunes image and networks by itself

I noticed that you have the docker run command with --rm option. In the Docker docs it says:

If you set the --rm flag, Docker also removes the anonymous volumes associated with the container when the container is removed. This is similar to running docker rm -v my-container

Perhaps this is why the volumes are being deleted?


What are known useful and misleading memes in the DevOps culture?

Useful memes:

"Wall of Confusion" from https://www.slideshare.net/littleidea/agile-infrastructure-velocity-09
C.A.M.S. from https://blog.chef.io/2010/07/16/what-devops-means-to-me/
"10+ deploys per day" from https://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr/ and https://youtu.be/LdOe18KhtT4
I.C.E. from http://radar.oreilly.com/2015/01/devops-keeps-it-cool-with-ice.html
"CI Theatre" from https://www.gocd.org/2017/05/16/its-not-CI-its-CI-theatre.html

There are some horrible memes pushed by vendors worth mentioning.

WARNING: Bad memes to use for explaining DevOps (traps for young players) -

"BI-Modal IT" by Gartner - criticised by Jez Humble here: https://continuousdelivery.com/2016/04/the-flaw-at-the-heart-of-bimodal-it/
"Multi-Speed IT" by Accenture https://www.accenture.com/us-en/insight-calibrating-multi-speed-it - (same as bimodal, but different name because of copyright - asked about here - What is good and/or bad about Accenture's Multi-speed IT approach to DevOps?
"The Three Ways" by Gene Kim https://itrevolution.com/the-three-ways-principles-underpinning-devops/ - the comments on the article include plenty of criticism


What are DevOps & Ops user stories, or elements in scrum context?

Make sure that your stories are not centered on Operations.  Remember that DevOps is a culture not a role.  Is an "Operations Engineer" really the stakeholder for the story?
Think of the features and business value they provide.  As a stakeholder I want feature so that business value. If you are struggling to identify the business value why are you doing it?
Stories are supposed to be "vertical slices" as opposed to "horizontal slices".  This means that stories should focus on an entire feature, not a layer.  This fits very well in a DevOps culture as your stories can cut across the traditional Development and Operations layers.

You already have some interesting features such as the quality standards. Now you just need to identify who the end stakeholder is, what the business value of this would be, and ensure that you write the story in such a way that people with development, operations, and possibly QA skill sets can work on it.


Monitoring immutable infrastructure virtual machines

I've looked a bit at Consul and Prometheus and am thinking that that is the way forward?

In short, yes.

Consul is designed exactly for this - it will run health checks against all of your services that you define and can detect when things go down. However, what it does not do it alert when things do go down, it is more designed for automatically taking things out of circulation when they are unhealthy. But it does give you a single point you can query for the health of your infrastructure which you can then alert on manually somehow or combine it with consul-alerts to do this for you.

Prometheus is geared towards metrics and can monitor the health of your applications in ways that consul is not really well suited. It has its only able to alert when defined events occur and you can tune it to monitor exactly what you require.

You might also want to consider external checks like what services such as panopta offer  (this is just an example, there are many alternatives out there) to check public endpoints to make sure they are still working from an external perspective.


What are best practices for sharing Docker compose files with ops team to use in Docker stack

Some thoughts:

You do not have to clone the entire project.  https://stackoverflow.com/questions/2466735/how-to-checkout-only-one-file-from-git-repository-sparse-checkout?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa discusses how to checkout a single file.  I have also had good results using curl to get single files off github.
You can containerize it.  Like this

FROM docker/compose:1.21.2
RUN mkdir /src
WORKDIR /src
COPY docker-compose.yml /src/docker-compose.yml
ENTRYPOINT ["docker-compose"]
CMD []

You can containerize docker-compose just like any other application.


Automation and the .NET framework which tools to use?

Somewhat stealing from Ian Margett's answer as the architecture is common amongst most Microsoft/.NET development organizations, the high-level target operating model for looks something like this:

The goal is to create a Continuous Deployment pipeline, using existing off-the-shelf software, namely TeamCity, ProGet, SonarQube and Octopus Deploy:

GitHub is the source code management tool, however, it could be BitBucket or Visual Studio Team Services.  The branching model and code review process are out of scope at this high level.
TeamCity is chosen as the Build System due to its tight integration with Octopus Deploy and good all-round support for .NET, msbuild and PowerShell.  TeamCity is also used as the orchestrator of deployments in Octopus Deploy.
ProGet is the package management solution which both stores Octopus Packages and proxies public package/image repositories.  The rationale for not using the built-in TeamCity NuGet store is purely for scalability reasons.
SonarQube provides continuous code quality management and reports are published as part of the TeamCity build outputs.
Octopus Deploy is used as the deployment tool for both infrastructure and code into the target platforms.

I've seen this broad approach implemented across two companies and successfully implemented it in two additional companies - in the most recent case we swapped TeamCity out for AppVeyor which worked, albeit a little painful when setting up firewall rules.


What Key Performance Indicators (KPIs) are used to measure DevOps?

I don't think there are any "universal" DevOps KPIs.  For example, velocity is great, unless it's not a key driver for your business.  Amazon cares a lot about velocity because they have a massive retail operation.  That's less important for a small app with 100 users.

This begs the question: how do you select the best KPIs relevant to your business?  That's a research and discovery process that involves your entire Enterprise.

What do you care about?

Quality
Reliability
Maintainability
Velocity
Process Improvement
Service Levels

What keeps your business stakeholders up at night?  What determines whether you make money this quarter or not?  The list above might include some of those things, or it might not.  Make your list, then figure out how to align incentives across every department to achieve them.

Incentives drive behavior, so decide collaboratively on SMART goals.  Pick two or three items off your brainstormed list, and start a measure/fix feedback cycle for each.  Don't pick too many at once- you're more likely to succeed by focusing hard on two or three things.


How to find interdependencies between pods in a Kubernetes cluster?

Have a look to solutions implementing a service mesh.

For example ISTIO, Consul, Ambasador + Envoy...


How can continuous integration scale for very large projects/teams?

Yes, but it's by preventing the things you mention in your question as much as possible.
You are right that there are only so much developers that can work on the same code before things become unmanageable. You need someone or something that has an overview of all changes and makes sure integration works fine and regression doesn't occur. This is hard to scale if everything happens in one repository.

Before starting any work you need to separate functionality in such a way that things can be integrated later with little or no effort.
This means that you want to split functionality into parts that are independent, or at least have as little dependencies as possible.
How you do this depends very much on the product you are developing. In the past separation of functionality was done by creating different libraries. The downside to this is that you need a good method of managing those libraries. Old versions of Windows for example had a bad library management system which caused the infamous DLL Hell.
Nowadays micro-services is (becoming) a popular way to do this, although this also has it's drawbacks (e.g. communication overhead, more services to monitor)

If separation based on functionality is difficult, another option may be to separate development in time. Instead of multiple teams working simultaneously on many different features, you have 1 team that works on only 1 or a few features at a time.

When separation of work is not possible, you'll need tools and tests that make sure things align. Automated unit and integration testing can be a big help here, but ultimately there will be limits to how big a project or team is to be workable.


How to trigger Jenkins Pipeline by push in specific branch in Bitbucket?

While I don't know Bitbucket-specific solution, when it comes to triggering Jenkins builds, I prefer using a simple post-receive Git hook.

In Jenkins select Trigger this build remotely (or something similar, it's usually the very first option in Build triggers section), then set up the post-recevie hook to something like:

#!/bin/bash

username=user
token=1234567890abcdefgh

echo "Executing hook"

while read oldrev newrev refname
do
   branch_received=$(git rev-parse --symbolic --abbrev-ref $refname)
   echo "   /==============================="
   echo "   | PUSH RECEIVED"
   echo "   | Old revision: $oldrev"
   echo "   | New revision: $newrev"
   echo "   | Reference   : $refname"
   echo "   | Branch name : $branch_received"
   echo "   \=============================="
   curl -X POST -d "BRANCH=$branch_received" http://$username:$token@jenkins/job/job_name/buildWithParameters?token=$token
   res=$?
   if test "$res" != "0"; then
      echo "The curl command failed with: $res"
   fi
done

exit 0

Of course, the above runs for every branch and passes the branch name as a parameter to the job. In your case, since you have different job for every branch, you would call different URL depending on what branch was pushed and probably use build rather than buildWithParameters endpoint.


How to assign static ip in docker for Windows for linux container?

You need create user define network

docker network create -d transparent --subnet=10.1.1.0/24 --gateway=10.1.1.1 TransparentNet3

Then you need run container with key --ip

docker run -it --name nano03 --network=TransparentNet3 --ip 10.1.1.1 someimage:sometag


What's the difference between "Ops Dev" and "DevOps"?

I've never seen ops dev used before this post.  Searching around I see things like this article that has it in the title but nowhere in the article.  Then there's this article from 2015:

  So in this way, when “OpsDev” happens, IT ops teams are helping developers do their jobs, but developers are also architecting applications with IT ops in mind, helping them utilize resources better. The days of developers simply creating applications and “throwing them over the fence” to testers and IT ops teams are over.

which seems to define OpsDev as "development in a company that practices DevOps" (or more cynically, "development done correctly").

Finally, there's https://dzone.com/articles/opsdev-is-coming :

  What Is OpsDev?
  OpsDev means that the dependencies of the various application components must be understood and modeled first before the development process begins. In addition, the consideration for infrastructure stability, environment modeling, security and audit/compliance measures are first and foremost. Application components are stubs and they do not need to be in their final forms. Secondly, the environments in which the components will be deployed for production must be modeled. Thirdly, the processes to deploy the various components to the target environments must be automated as much as possible.

Again, this sounds to me just like normal, modern development.

DevOps is a problematic term because it has many different definitions.  OpsDev, on the other hand, appears to have essentially no definitions.  From a personal perspective, I would mentally substitute "development" for every instance of "OpsDev" and ignore the term; it appears to be just another instance of people inventing new terminology to describe old concepts so that they sound more exciting.


File transfer from Application server to Db Server without a shared folder access but using Service Account

There is a general lack of understanding here.  I suspect your company doesn't understand your "requirement" and the Server team has provided something that either won't do the job or your team don't know how to make it work.

You say that you "need" to get a file onto a Database server - my first question would be - "why"?

What's in this file that you need to put it anywhere near the Database server?

Your company has a Policy against doing this through file shares on the Database server.  [Being a DBA] I completely understand (and agree) with this Policy.

On its own, a Service Account isn't going to be much use to you.
You might be able to use it with some File Transfer utility to copy the file but, again, why do you want to do this?

Take a step back and describe the "Big Picture" problem that you're trying to solve here.


I had this problem when I needed to build something at home.

What I did:

installed docker in a Ubuntu server VM
installed docker in a windows server 2016 VM (if you need windows containers like i did)

THEN:
In the Ubuntu machine:

I created a local registry running in a container (1 command :)  )
I spin up a container with a git server - I "linked" the configuration folder to a folder in my machine for persistence.
I spinn a container with jenkins server (same trick with the folder)
I spin up containers with slaves
I have a database container attached to a nexus container for artifacts and build dependencies

In windows VM :

I spin a jenkins slave container to build windows apps
I spin IIS machines to test the built app

My home computer is 32 GB ram / core i7 and it works great , even with windows container databases (these are monstrous)

Tricky for me - work in progress - build a container in a slave running in a container . It can be done but I am on googling phase.

Advantages:

if you use docker compose and you save your containers in the local registry you get to move your work easily . I always version my docker compose files in a github repo.
you can pause the VM host and move everything in some other host.
you can branch your things for having a test env by cloning the machine and copying the host folders


Jenkins shows the job as failed if there is nothing to commit to gitlab

In order to get a success, you need to avoid non-zero exit codes. A simple solution would be to change the last line of your script to git commit -a -m 'Changes pushed by Jenkins' || true, but better solutions would parse the output of git-add and only run git-commit when there is something to commit.


What is recommended books reading order offering thought leadership to read on DevOps to go from beginner to expert?

One fantastic read that offers great thought leadership on this issue and is specifically tailored to the non-technical is The Phoenix Project which is available in paperback, a Kindle E-book and as an Audiobook. This book offers an excellent starting place to educate on the who, what and why of DevOps, wich is often seen as nebulous buzzword. The official synopses states,

  Bill is an IT manager at Parts Unlimited. It's Tuesday morning and on his drive into the office, Bill gets a call from the CEO.

  The company's new IT initiative, code named Phoenix Project, is critical to the future of Parts Unlimited, but the project is massively over budget and very late. The CEO wants Bill to report directly to him and fix the mess in ninety days or else Bill's entire department will be outsourced.

  With the help of a prospective board member and his mysterious philosophy of The Three Ways, Bill starts to see that IT work has more in common with manufacturing plant work than he ever imagined. With the clock ticking, Bill must organize work flow streamline interdepartmental communications, and effectively serve the other business functions at Parts Unlimited.

By novelizing this story the authors are able to expertly illustrate and explain in a detailed and entertaining manner:

What DevOps means and entails
What teams and individuals are involved in DevOps
How DevOps can benefit a companies' workers and business
Challenges and weaknesses of the traditional model

While this will not explain or recommend how to do DevOps on a technical level, it will make and explain the business case and method for doing so, thereby taking you from beginner to intermediate.

The authors have also published a followup tome, the DevOps Handbook which bills itself as a how-to on "Creating World-Class Agility, Reliability, and Security in Technology Organizations," however one reviewer, FA Calkins cautioned,

  Dont buy this book if you expect it to teach you how to build a CI/CD DEVOPS environment. It is basically a review/summary of a bunch of research papers that the authors have read, but does not provide any specifics details of how to set up a CI/CD pipepline (only theory). You can get most of the content of this book by reading articles from the web. Also, you can get more specific details of how to implement CI/CD with specific tools, programs, and scripts from other sources. Dont waste your time trying to read this book if you want to deploy CI/CD.

Though I have not yet read it (it is on my reading list!), the description claims that the book includes

  More than 40 DevOps case studies, including Amazon, Etsy, Capital One, Google, Facebook, Intuit, Nationwide Insurance and many more
  More than 400 pages of DevOps applications, lessons and 'how-to’s'.
  DevOps data gathered from more than 25,000 data points.

These resources should at least get you started on the business/philosophy piece.


Bitbucket pipelines: Deployment destination dependent on commit author

the idea is for each team member to have his/her own environment assigned, but deploy using a shared parametrized pipeline

You could use definitions for the steps for each environment.

definitions:
  steps:
    - step: deploy to staging1
        script:
          - echo "Staging 1"
    - step: deploy to staging2
        script:
          - echo "Staging 2"

Then just express each colleague branch and restrict his commits to just this branch.
For each of the branches you could call your needed definitions.

branches:    #these will run on every push of the branch
    feature/john.b:
      - step:
          # call needed definition step/s
    feature/pavel.m:
          # call needed definition step/s

This gives u flexibility to express multiple definitions steps for one person if needed.


A CIO is part of the company leadership. His primary duties is to make policies, create organization (i.e., structure the company, at least the part that reports to him), make strategic decisions etc. In a large company, many technical people report directly or indirectly to the CIO; if there are technical problems to solve, he is supposed to hire people to solve those problems, or - more likely -
have his direct reports hire those people.

I would be very wary if a CIO starts fixing technical problems himself. This probably would mean that he has trouble delegating, or trusting his employees.

There may be a phase in a very small company (like 5-10) where you might have some kind of senior technical person who starts to grow into the CIO role. But that should certainly be a path that develops itself (often it is quite obvious who is a good fit for that role and who isn't), and nothing where you start pondering whether you want to become CIO or not out of the blue.


ELK - micro service and network latency monitoring

ELK stack does not provide monitoring. It only provides reporting unless you configure additional add-ons such as ElastAlert.

ELK also only reports (or alerts if configured) on the data that you feed into it. So to turn the question around:

  Do you have your network devices configured to send micro service and network latency logging information to your ELK stack?

If yes, then the answer will be yes. If no, then the answer will be no.

ELK is nothing more than a system for processing and reporting metadata and generating structured data from plaintext logs, so it will do whatever you configure it to do.


Bamboo doesn't have a color status like Jenkins per say, but it does have a very similar build status indicated by a circle with a symbol within it. This status is available for both the entire build plan and for an individual build.

For your GET request, you will want to use the same api calls that you are using to get the buildstate value to get a SUCCESSFUL or FAILED state.


How can I clean/delete a Jenkins workspace after merge?

I am assuming you are using Jenkinsfile. You can do for example:

pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                echo 'Building..'
            }
        }
        stage('Test') {
            steps {
                echo 'Testing..'
            }
        }
        stage('Deploy') {
             when { branch 'master' } /* when in master branch */
             steps {
                echo 'Deploying....'
                deleteDir() /* clean up our workspace */
            }
        }
    }
}


how to give a user privilege to run Ansible commands or just an specific tag?

There is no tool in Ansible to authorize a user to execute selected commands or tags only.

It seems a good idea to use sudo to authorize user to execute selective commands.


How to prevent group from deploy to environment but allow same to redeploy?

You can add a post deployment aprroval to the dev environment that the dev team has to approve.
And then you can add a pre deployment approval to the QA environment that the QA team has to approve.

Then a release will be automatically triggered for the QA environment after the dev team has approved the post deployment approval on the dev environment. But it will not deploy to the QA environment until the QA team has approved the pre-deployment approval.

Additionally, the QA team can re-release any previous release to the QA environment at will by going back to the release and clicking redeploy.


Bare-Metal Kubernetes storage in enterprise

A storage solution for kubernetes is a pain for those of us who chose to run kubernetes on premises.

What storage solution one picks would certainly depend on potential usage, a solution that will perform well for simple file operations would not pan out very well if you want to run a database in kubernetes.

Given your configuration, I think "vSphere Cloud Provider" is your best option

  VMware offers a Cloud Provider known as the vSphere Cloud Provider (VCP) for Kubernetes which allows Pods to use enterprise grade persistent storage.

architecture overview of vSphere Cloud Provider

If you don't want to run databases you can use NFS using HDFS setup you have or any other NFS server.

Disclaimer: I have not used vSphere Cloud Provider or ceph myself. Personally, I have used NFS from our storage server for production and it works fine for moderate database loads. We also have a large gluster implementation supporting close to thousand volumes, however, based on issues we have seen with gluster I will not suggest that to anyone.

On a side note, you don't "have" to use all of your cluster nodes as "storage nodes" for Ceph & Rook. Only storage nodes will have restrictions on them. If you are going with Ceph & Rook, I will suggest to use dedicated storage nodes otherwise it may impact performance of your workloads.


Docker can either run the command using the equivalent of the OS exec syscall directly in the kernel, or it can run a shell (typically /bin/sh) to call your command. A shell provides functionality you are accustom to from the command line, including:

I/O redirection and pipes (>, >>, <, etc)
Command chaining (&& and ||)
Variable expansion ($var)

However, a shell may also intercept signals, particularly when running as pid 1, which can prevent a graceful shutdown of a container (not much of an issue for a healthcheck). In addition, to run a the shell syntax, your container's image must include a shell, which means that won't work for images based on scratch that do not include a /bin/sh.


To design a blue green deploy how to publish websocket traffic from live to hot swap server

Websockets don't really have any handling for this per se. You could build your own system but generally the idea for HA is that if anything happens you'll set the client up to automatically reconnect. There will be some non-zero downtime during a blue-green swap but if your app is well built it should handle that already.

If you wanted something more smooth, you could build a message in to your protocol to connect to a new server and then drop the original connection.


How to automate the process of patching Jenkins Slave instances in AWS

If you use the Self-Organizing Swarm Plug-in to connect your build agents to your master, you can spin agents up/down without having to make any changes on your master. Agents will automatically register themselves with the master. You will have to supply credentials to the agents to self-register. I would suggest using SSM Parameter Store to store them and then retrieve them in userdata.

Your agents can then be completely ephemeral, and updates performed in userdata or by updating the AMI.

See this Terraform module for a more detailed example of implementing this.


Why does Jenkins Pipeline script returnStatus value not match status examined in shell?

There's a few things to note. First is that in the absense of a shebang being supplied, the actual interpreter that will be used is actually "sh", not "bash".
Taking a look at the Jenkins source for the BourneShellScript task, what happens is that Jenkins does not actually return the exit code from the subshell - it monitors a file called "jenkins-result.txt for changes (using a FileMonitoringTask) and when that file has been updated, it merely assigns the content of it as the exit status.
From what I can see, when you call "sh" it does not just run your script code, but actually wraps it in a fairly substantial block of boilerplate code (refer line 272 of the above linked BourneShellScript). In effect, something closer to this:
sh -c ({ while [ -d '/home/user/workspace/newbie_dont_know_what_hes_doin@tmp/durable-2fa119e0' -a \\! -f '/home/user/workspace/newbie_dont_know_what_hes_doin@tmp/durable-2fa119e0/jenkins-result.txt' ]; do touch '/home/user/workspace/newbie_dont_know_what_hes_doin@tmp/durable-2fa119e0/jenkins-log.txt'; sleep 3; done } & jsc=durable-2fa119e0; JENKINS_SERVER_COOKIE=$jsc sh -xe '/home/user/workspace/newbie_dont_know_what_hes_doin@tmp/durable-2fa119e0/script.sh' > '/home/user/workspace/newbie_dont_know_what_hes_doin@tmp/durable-2fa119e0/jenkins-output.txt' 2> 'jenkins-log.txt'; echo $? > '/home/user/workspace/newbie_dont_know_what_hes_doin@tmp/durable-2fa119e0/jenkins-result.txt.tmp'; mv '/home/user/workspace/newbie_dont_know_what_hes_doin@tmp/durable-2fa119e0/jenkins-result.txt.tmp' '/home/user/workspace/newbie_dont_know_what_hes_doin\@tmp/durable-2fa119e0/jenkins-result.txt'; wait) >&- 2>&- &

This is likely necessary to support scenarios where the node on which the script is executed is not the node on which the pipeline is executing, however it does leave a lot of room for error in returning the result, depending on the idiosyncracies of whatever interpreter "sh" is aliased to (not necessarily bash) and the OS on which it is running.


is it possible to connect an AWS autoscaling group with an application load balancer?

Your CloudFormation template currently has the following resources:

A Launch Configuration
An Auto Scaling Group
A Load Balancer

Follow these steps in sequence to attach the auto scaling group to the load balancer:

Add a listener resource to your CloudFormation template, as described here. Set its LoadBalancerArn property to the load balancer's ARN.

Add a target group resource to your CloudFormation template, as described here.

Add a "listener rule" resource to your CloudFormation template, as described here. Set its ListenerArn property to the listener's ARN. In its Actions property, add a forward action & provide the target group's ARN like this:

Actions:
- Type: forward
  TargetGroupArn: !Ref MyTargetGroup

Add the target group's ARN to the TargetGroupARNs property of the auto scaling group, as described here.

If you prefer to visualize a chain of CloudFormation resources linking the load balancer to the auto scaling group, it'll look like this:

Here's a complete example: LB to ASG Example.


Error Vagrant Provisioning with Virtual Box on Fedora

Try to start with a clean sheet by removing the .vagrant.d folder or start to remove the corrupted zip. When the file or folder has been removed, one could run vagrant up again and check whether it works.


If you are running on VMs or outside of a cloud provider then manually setting this up involves several steps.  The tool you will need is kubeadm (see documentation linked) and you will need to init the nodes and then kubeadm join the nodes to the master.

If you are using a cloud provider, then follow the providers documentation on how to add additional nodes.

A great guide called "Kubernetes The Hard Way" is another good resource for learning how to setup a cluster without any automated tooling.


Gitlab - Is there any reason not to default enable 'Allow Users to Request Access' for Projects?

It's just a safeguard for preventing unwanted user accounts from accessing projects. It's disabled by default so that you can control from the start which accounts can access a project and not have unwanted spam of requests for access.

One example I can think of, is if you only wanted users to be able to use their company email/account to access projects, and not their personal account. It could even happen by accident that someone unknowingly requests access with the wrong account.


Will google cloud take money from me in this situation?

Yes, they mean the Compute Engine. So you pay Google only for API, not for server.


Dockerized node.js and bind mount permission problem

In your Dockerfile:
ARG UID=1000
ARG GID=1000

RUN usermod -u $UID node && groupmod -g $GID node

Then using docker build:
docker build --build-arg UID=$(id -u) --build-arg GID=$(id -g) .

The ARG lines provide defaults. If you leave off the --build-arg flags, they will be used. If you do use the --build-arg flags values those will be used instead.


How to get information about resources in an Azure Pipelines run?

You can do:
variables:
  templates.ref: $[ resources.repositories['templates'].ref ]

See Repository Details docs.


loop requires a valid list. But we got here is,

[({'i_am': 'sam', 'eggs_ham': '456'}, AnsibleUndefined),
 ({'i_am': 'sam', 'eggs_ham': 'ham'}, {'transport': 'train'}),
 ({'i_am': 'eggs', 'eggs_ham': '456'}, AnsibleUndefined)]

Since no value is assigned to transport_fact_results when common_thing_2 is not defined.

if you remove the when condition it will work.

CODE:

- hosts: localhost
  vars:
    default_thing_1: 123
    default_thing_2: 456
    default_transport: train
    list_of_things:
      - name: foo
        common_thing_1: "sam"
      - name: bar
        common_thing_1: "sam"
        common_thing_2: "ham"
      - name: biz
        common_thing_1: "eggs"
  tasks:
    - name: Set the Things Facts
      set_fact:
        thing:
          i_am: "{{ item.common_thing_1 | default(default_thing_1) }}"
          eggs_ham: "{{ item.common_thing_2 | default(default_thing_2) }}"
      loop: "{{ list_of_things }}"
      register: things_fact_results
    - name: Set Optional Things Facts
      set_fact:
        thing:
          transport: "{{ item.transport | default(default_transport) }}"
      when: "item.common_thing_2 is defined and item.common_thing_2 == 'ham'"
      loop: "{{ list_of_things }}"
      register: transport_fact_results
    - name: Debug the Facts
      debug:
        msg: "{{ item.0 | combine(item.1|default({})) }}"
      loop: "{{
        things_fact_results.results |
        map(attribute='ansible_facts.thing') | list |
        zip(
          transport_fact_results.results |
          map(attribute='ansible_facts.thing') | list
        ) | list
      }}"

OUTPUT:

PLAY [localhost] ************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Set the Things Facts] *************************************************************************************************************************************************************************************
ok: [localhost] => (item={u'common_thing_1': u'sam', u'name': u'foo'})
ok: [localhost] => (item={u'common_thing_1': u'sam', u'common_thing_2': u'ham', u'name': u'bar'})
ok: [localhost] => (item={u'common_thing_1': u'eggs', u'name': u'biz'})

TASK [Set Optional Things Facts] ********************************************************************************************************************************************************************************
ok: [localhost] => (item={u'common_thing_1': u'sam', u'name': u'foo'})
ok: [localhost] => (item={u'common_thing_1': u'sam', u'common_thing_2': u'ham', u'name': u'bar'})
ok: [localhost] => (item={u'common_thing_1': u'eggs', u'name': u'biz'})

TASK [Debug the Facts] ******************************************************************************************************************************************************************************************
ok: [localhost] => (item=[{u'eggs_ham': u'456', u'i_am': u'sam'}, {u'transport': u'train'}]) => {
    "msg": {
        "eggs_ham": "456",
        "i_am": "sam",
        "transport": "train"
    }
}
ok: [localhost] => (item=[{u'eggs_ham': u'ham', u'i_am': u'sam'}, {u'transport': u'train'}]) => {
    "msg": {
        "eggs_ham": "ham",
        "i_am": "sam",
        "transport": "train"
    }
}
ok: [localhost] => (item=[{u'eggs_ham': u'456', u'i_am': u'eggs'}, {u'transport': u'train'}]) => {
    "msg": {
        "eggs_ham": "456",
        "i_am": "eggs",
        "transport": "train"
    }
}

PLAY RECAP ******************************************************************************************************************************************************************************************************
localhost                  : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0


Continuous integration using bitbucket pipelines failed for OpenEdx

The author of the Q&A indicated that the issue has been solved since absolute paths were used.


Is this okay to use double proxy-server setup (on host and in container) on one machine?

Sure, you put in as many proxies as you need. You don't want to over-complicate things, but it seems you have a good reason for your architecture.


On Premise CI / CD recommendations for Docker images

Jenkins Option

If you're comfortable with Jenkins and want to use a CI tool, it is a decent option.

It has multiple plugins for watching for files arriving in a file system, and at the end of the day, you can even just wrap your bash scripts into it so you have a central GUI and recorded logs.  So, you're not limited in any way by it.

Inotify Option

I personally might just have a CI tool like Jenkins or GitLab-CI deploy a bash script though.  Then you could use inotify to properly watch a folder and execute your script exactly when things arrive.  See: https://www.howtogeek.com/405468/how-to-perform-a-task-when-a-new-file-is-added-to-a-directory-in-linux/ for an example.

Why iNotify?

In many tools, I believe you generally have to "poll", which means run over and over just to see if you have anything to do. Inotify lets you genuinely "watch" and be pushed events only when things happen.  This is cleaner and more efficient as you won't have hundreds or thousands of tasks running a day that literally do nothing.

Monitoring

Note: One plus side to the pure Jenkins approachis built in monitoring.  Every  job run that fails can easily email/alert you.   You can pull this off with the iNotify script as well, but  if you don't have the right tooling for monitoring/etc in your environment, you may favor the Jenkins approach.


How to not "fire and forget" when doing a remote trigger of a Jenkins build?

I've created the jenkins-remote-builder script which will follow your remote build to completion.
Some more details about it (from its README.md):

Example
jenkins=https://user:pass@jenkins.mydomain.com:8080
jenkins_job=MyApp-Deploy

environment=dev
application=myapp
revision=9fd71f63b351b8208264daf86d292ced580a2f60

./jenkins_remote_trigger.sh \
            -h ${jenkins} \
            -j ${jenkins_job} \
            -p "ENVIRONMENT=${environment}&APPLICATION=${application}&REVISION=${revision}"

Usage:
-h HOST     | --host=HOST                       Jenkins host
-j JOBNAME  | --jobname=test-build-job          The name of the jenkins job to trigger
-p JOBPARAM | --jobparam=environment=uat&test=1 Jenkins job paramiters
-q          | --quiet                           Don't output any status messages


How do fail a Packer pipeline upon powershell provisioner pester test failure?

After resolving some syntax issues, I found the true cause of failing to throw the error.
In my prior usage I was doing this through Inspec provisioner, which handled the exit codes.

If you are using Pester and having to do the workaround I described above, then to ensure error's in a step are thrown you need to throw an exit code other than 0.

The two steps I took that seem to work fine (I like some emoji fire in my logs )

$ErrorActionPreference = 'Stop'

try
{
# Do code here
}
catch
{
   Write-Error " Failed to do something important `$TestFile: $TestFile"
  exit 1  # packer will recognize failure at this point
}

Note that you will want to make sure your Azure DevOps Pipeline has the following settings to publish test results regardless of failing:

# ℹ  Using Invoke-Build to call Packer. You can do this with a packer extension or any other way you prefer
- task: PowerShell@2
  displayName: Run Packer Configuration
  inputs:
      filePath: build/build.ps1
      arguments: '-Task PackerBuild -Configuration $(Configuration)'
      errorActionPreference: 'Continue'
      pwsh: true
  continueOnError: true #I want to publish the test results regardless

- task: PublishTestResults@2
  displayName: Publish Pester Tests
  inputs:
      testResultsFormat: 'NUnit'
      testResultsFiles: '**/TEST-*.xml'
  condition: succeededOrFailed()


No. Because you cannot distribute Oracle JDK or JRE, the license in effect doesn't allow distribution. When distributed by a third party (embedded with your app) all the liability for it not working is on that party. This is why you will not find Oracle JDK/JRE on any of the public Docker registries, or in any Linux package repositories for that matter.

You can create your own image, and install Oracle JDK or JRE using the webupd8team/java package for Ubuntu/Debian.

And if you are using Alpine Linux this blog post explains which dependencies are required, and links to StackOverflow for installation instructions.

Update: Oracle has an official Docker image on the Docker Store now. https://blogs.oracle.com/developers/official-docker-image-for-oracle-java-and-the-openjdk-roadmap-for-containers

Link to Docker Store Oracle Java Image - https://store.docker.com/images/oracle-serverjre-8


What is the name of Jenkins plugin for GitHub plugin configuration?

The plugin that you are looking for is: https://github.com/jenkinsci/github-plugin.

You need to go in Manage Jenkins > Global Configuration to find the GitHub plugin configuration (according the documentation).


Vagrant is for setting up test environments right? Ansible/Chef etc. seem good for servers you own, but what about cloud? Docker... great for standardizing app, so why use Ansible/Chef? Kubernentes.. good for deploying to multiple environments and scaling

Just a personal note: the way you formulate these things makes me assume that you are diving freshly into these tools, and trying to figure out what they are for. And your reasoning is "tool X is for purpose Y". I know the feeling, I started out like that as well, trying to wrap my mind around what all this stuff is for. The trap is that the real reasoning should be "tool X does Z", instead. Few of these are really limited to a particular purpose.

For example. You say "Vagrant is for setting up test environments". This is wrong. Vagrant is for spinning up VMs in an automated way, including a zoo of pre-made images. "Ansible is good for servers you own" is wrong. Ansible is a tool to execute commands on a remote server where you have ssh access, with a zoo of recipes/playbooks to do the most common tasks. And so on.

This is not to berate you, but to make it easier for you. Try to be a little less abstract, and just look at what the tool actually does. Then it gets relatively easy to find proper uses, and you also can use the same too for different purposes.

For example:

Vagrant is great for also setting up development environments, not just testing. Or, really, any VM at all where you want the actual content to be mostly configuration-as-code and want to avoid clicking around in a VM GUI.

  I'm just trying to understand what the need for ansible or chef are? And how about Vagrant?

Now you know what the need for Vagrant is.  Vagrant literally solves the need to spin up VMs repeatably, automated, and have the configuration managed as files (i.e., checked into VCS together with the source code of your application, and so on).

  Also for the build cycle CI/CD pipeline with Git and artifacts.

Read the documentation of git & co and actually look at what it does. That, then, is its purpose, nothing more and nothing less.

Same for CI/CD. Continuous integration means literally that. To continuously (i.e., once or more a day, not just every few weeks or months, as it was done in the past) integrate (i.e., merge all the branches of your application that are in development currently). There is no hidden purpose there (or if there is, it should be obvious - doing it more often means every single time is more easy, since there's not such a long time inbetween). Obviously the details of how it works exactly without turning into a nightmare could be more involved, but the purpose is just what it says on the can.

  AWS, GoogleCloud, Azure ? Which is best

It does not make sense to think of "best" here. All three are extremely complex, and the fact that they are all still around means they all serve a purpose. You have to look at each of them, try them out, and compare them to your needs (and your personal likes/dislikes, sometimes).

  Any recommendations about the best overall Devops setup?

Clearly the best overall DevOps setup is the one that solves your problem and fits your overall needs and wishes. There are a vast amount of possibilities of good DevOps setups, and you have to choose for yourself.


Make TeamCity see test results in real time using Maven, JUnit 4 and Surefire

So I have choosed implementing my own RunListener. I added a compile,test dependency on it to all my other modules and hooked it up in maven surefire plugin configuration. See also https://stackoverflow.com/questions/61206966/surefire-only-reports-first-test-suite-from-module-to-runlistener


Docker Images via Container vs Dockerfile

If one uses a Dockerfile then a colleague could also understand what happened (documentation as code). If one runs a container, enters it, runs commits then it would be hard to understand what packages were installed. Especially after a couple of months.


These are really the pitfalls as running a VM or traditional server with 32 bit CPUs.  Most notably is memory addressing.  32 bit CPUs can only access up to 4 gigs of memory.  This isn't something i would expect most containers have an issue with as they should be single, small, light applications.  One thing that you might run into is any binary built targeting specifically 64 bit architecture will not be able to run.  This is any application and their DLLs.  I don't think any web API will have issues with this, you will need to be aware of console applications.  Most dot net applications target any cpu allowing the runtime to target the correct underlying architecture.  There are applications that do target specific architectures and you will need to be aware of which ones they are.

In today's world, storage space is cheap.  The exception to this rule is embedded development (but I don't think that is the case here).  You need to ask yourself, is the extra 4% saved space worth the cost of me managing and ensuring that all the applications i'm running work on a 32 bit architecture.  Remember 32 bit applications will run on 64 bit architecture, but not the other way around.  Unless there is a specific need to only run 32 bit, like testing an emulated embedded environment (which is only 32 bit), I would just go with the 64 bit versions of everything.  To me, using 32 bit is solving a problem you don't actually have.


(N.B. this answer is, in the spirit of the question, high-level and not very concrete...)

  How can I calculate throughput of our system

If by "calculate" you mean to take some data points, dry run some formulas and get a pretty close approximation to your throughput - that is pretty much impossible, unless you have a lot more information - which may also include pretty random parts, hence quite hard to do theoretically.

If you mean "measure", then it could be as simple as checking your logfiles and counting lines per time unit.

  and build monitoring around max load that our system can handle?

Unless you're NASA, that's probably a case for trial and error. Run your system(s) and see how much troughput you get. Increase worker nodes. See what happens.

If you already know the behaviour of your overal system - i.e., whether it scales linearly, whether there are bottlenecks like databases, lock contention and such, then you can take shortcuts or good guesses.

That's one of the reasons for doing the container-based scaling we do these days - you can throw a few more workers at it relatively easily.

How you actually (technically) do that depends on what platform you are using. AWS, Kubernetes/OpenShift etc. come with techniques or settings do do it automatically for you. I assume those work mainly with the metric of "free workers" - trying to get the "free:busy worker" ratio into some target corridor so that every new request has a very high likelihood to hit a free worker, at any time, thus getting (theoretical) constant time for each request.


How does dependency management for Node.js differ from Maven?

Strange enough node.js unlikely maven installs the packages the software needs inside the project directory.

I don't see fundamental differences between npm on node.js and Maven. In both cases, the dependencies finally end up in the project directory (npm: /node_modules, Maven: /target). Otherwise, you cannot continue building the software. (When you run the build system to create the final artifact, it needs the dependencies.)

In addition, both have a global cache to avoid downloading already fetched dependencies: In Maven, the default cache directory is ~/.m2 (on Linux and MacOS). For npm, it is ~/.npm.

  Is there a way to introduce a central repository which also caches different versions of components and would allow to pull them on build like we know it from other ecosystems?

If the system cache is not enough and you do not want to depend completely on the public repository (npm: registry.npmjs.org, Maven: Maven Central), you can run your own mirror. For instance, if you want to be able to rebuild the software even if the public repository is down.

In my previous job, we used Nexus as a caching proxy and local registry (for hosting code that is only accessible within your organization). It supports both npm (see documentation) and Maven (see documentation). (There are other tools, but the point is that both ecosystems can be handled by one tool.)

Of course, there are differences between Maven and npm. For instance, Maven is also a full build system, whereas the scope of npm is primarily dependency management. Maven also defines conventions to structure a Java project ("The Maven Way"). But when looking only at the dependency aspect, package managers for Node.js like npm are conceptually not so different from Maven.


Multiple Application in single SSL 443 server NGINX

Oops, turns out I just have to remove the line
listen [::]:443 ssl ipv6only=on;
and it works.

it's leftover from running NGINX certbot, I thought that was important


How to check available volume provisioners in Kubernetes cluster?


Scale number of pods in OpenShift from oc command line

found the issue,  instead of 'dc' need to use 'deploy'

oc scale deploy name_of_deployment--replicas=2


Is it possible to consolidate notification handlers when they are very similar?

In ansible 2.7 and earlier versions, variables in handlers name are not populated.
This bug was reported and a correction was lately pushed to devel version.

You need to run ansible from devel, and hope they publish this fix in the next release.


Download Google Cloud Bucket as a file (zip)

You seem to be confused a bit about how Firestore stores data and what Google is suggesting you do.

  What is the problem? You are backing up a bucket inside other bucket.

This is not correct.  Firestore stores its data inside some sort of mostly opaque database format.  If you are copying this out to a GCS bucket, that really is an export into a different storage medium and a different format.

That being said, it's still reasonable that you might want to keep your data backups outside of Google (although GCS has a good SLA).  To do that, after you've exported your Firestore data to a GCS bucket, use the gsutil utility to copy that bucket down:

gsutil cp -r gs://your-bucket-name .


Add mapping ip and host into /etc/hosts with docker

You could try to use docker compose and the docker compose "extra_hosts" tag, as described on the docker compose specification at https://docs.docker.com/compose/compose-file/.

  extra_hosts

  Add hostname mappings. Use the same values as the docker client --add-host parameter.

  extra_hosts:

  "somehost:162.242.195.82"
  "otherhost:50.31.209.229"

  An entry with the ip address and hostname is created in /etc/hosts
  for this service, e.g:

  162.242.195.82  somehost

  50.31.209.229   otherhostinside containers

So, this will mean, that you need to use docker compose in addition to docker.


How is Continuous Integration different from using git to integrate external code?

Merging code changes is definitely handled by Git and its tools.  But Continuous Integration also includes testing and building the software. The development teams are provided with constant feedback on the status of their branch(es) and whether they are failing tests, builds, etc.

Jenkins provides the tools to enable this feedback.  By creating a CI "pipeline" in Jenkins the developer gain much better insight into the build, beyond what simple Git merges and stashes provide.


You'll notice that the official recommendations for directory layout put all of the playbooks at the root level.  This is intentional, as Ansible doesn't handle other schemes well.

You can put playbooks in a subdirectory, as you've started to do, but that will require (as you've already found) the use of relative paths from the playbooks to any other resources they use; Ansible uses the playbook path to start its search, that's just how it's written.  For the most part, that's just what you get to deal with when you make the choice to move where playbooks live.

However, you seem to have more nesting than you should:

---
- hosts:  aws_instance.jenkins-agents
  user:   ec2-user
  vars_files:
    - ../../../vars/main.yaml
    - ../../../vars/vault.yaml
  ...

Why are you navigating up three directories?  I'd expect a layout that's like the official one, but with one subfolder for playbooks; then you'd end up with just

---
- hosts:  aws_instance.jenkins-agents
  user:   ec2-user
  vars_files:
    - ../vars/main.yaml
    - ../vars/vault.yaml
  ...

which is a much more minor change.

Secondly, it's pretty rare that you actually want to use vars_files.  Most variables used in Ansible either vary based on the host (and thus should go in group_vars/host_vars in the project root) or are role-specific (and thus should go in vars/defaults in the role directory).  A few documentation links:

https://docs.ansible.com/ansible/intro_inventory.html#splitting-out-host-and-group-specific-data
https://docs.ansible.com/ansible/playbooks_roles.html#roles
https://docs.ansible.com/ansible/playbooks_roles.html#role-default-variables

Personally, I think it's best to have as little as possible in your playbooks, but delegate everything out to roles.  Here's an example entire playbook in your setup:

- hosts: aws_instance.jenkins-agents
  roles:
    - ../roles/jenkins_agent

This gives you far more flexibility for re-use.

And if you find yourself not wanting to use relative paths for roles, you can override roles_path in an ansible.cfg in the root directory where you run your Ansible commands from.


replacing values in shell script using sed

Negation to the rescue:

sed -i "s/server.address=[^\ ].*\ /server.address=${SERVER_ADDRESS} /g"

This is telling to search zero or more characters (.*) NOT matching whitespace ([^\ ]) up to the first whitespace.

A good pattern is not to use real values to be replaced (10.111.11.11:3197) but to use a mark like ###SERVER_ADDRESS###. If something goes wrong you can easily post-check with

grep "###" && exit -1

This can avoid continuing with dangerous values (mixing dev/pre/pro values)


docker-compose invalid type, it should be a string

The Docker Compose file reference states, that environment variables are defined as VARIABLE=value array elements. For your case, the docker-compose.yml file would need to be changed to this:

version: '3'
services:
  server:
    ports:
     - 13045:3000
    environment:
     - NODE_CONFIG='{"DATABASE_URL":"http://db:5984"}'


I need a good tool for running a series of Spring Boot based tools on Docker, where each needs a web page and a database connection

According to me a CI like Jenkins or gitlab should be used to build the docker images when the git code changes. Once this works well, you could start to run some scripts that deploy to an environment (CD). First of all, I would like to emphasize that implementing CI is the highest priority as the docker images seems to be built manually at the moment.


How do I package a jar and a Linux VM to run in Windows?

Generally people go the opposite direction and try to get more to run on Linux as it is generally cheaper (e.g. license free).  Also, Windows 10 is more of a personal workstation (vs Windows Server).  So, it seems like it may be a short term strategy (unless you're just learning on your own).

Having said that, the most DevOps standard way to do this would be to:

Install Docker on your Windows 10 PC.  Docker (in layman terms) lets you run apps on top of slimmed down operating systems of various kinds in a portable manner.
Dockerize the application(s) running on the Linux server.  You can start off with some slim linux variant like Alpine and just add what you need.
While you're at it, just Dockerize your windows 10 Java app at the same time and start running it from command line rather than NetBeans.
Now, you can run it all on Windows or Mac Or Linux (or kubernetes) with ease.

This assumes the server-side app is not a Java or otherwise portable app.  If it is portable, you can just run it locally and forget all this (although all of this is still a good idea).


What I would suggest, like comments under your post, is to take this to the #inspec channel on the Chef Community Slack to ask there. That said, here's how I might attack this:

control 'some-control-slug' do
  title 'My control'
  impact 1.0
  desc 'Example code for Stack Exchange'

  if os.windows? && ::Gem::Version.new(os.release) < ::Gem::Version.new('6.1')
    # Only executed on Windows machines older than 2008r2
    describe directory('C:/Users') do
      it { should exist }
    end

  elsif os.windows? && ::Gem::Version.new(os.release) >= ::Gem::Version.new('10')
    # Only executed on Windows Server 2016 or newer
    describe directory('D:/MyFolder') do
      it { should_not exist }
    end

  end

  # Executed on all host types
  describe sys_info do
    its('hostname') { should match(/SomeAwesomeHost/i) }
  end
end

Additionally, you can limit the entire inspec control with Chef-like hooks, such as only_if. The above example code, as written, may execute on a Linux target. We could limit all of them by placing the following up near the desc or title:

only_if { os.windows? }


What's are the advantages of using CloudBees Core over Jenkins?

I personally prefer Jenkins over CloudBees as we have enough Jenkins knowledge in our engineering department. However, if their is a lack of such knowledge or interest then it could be useful to use a Software as a Service (SaaS) product like CloudBees. In general, this means that the department will be unburden as the platform will scale automatically and the SaaS provider is responsible for keeping Jenkins up and running.

A downside of an Enterprise solution is that a fee is required. However, maintenance and in particular the number of resources that is required to apply it, is often underestimated in my opinion. Maintaining Jenkins by Engineers from the department means that they have less time to develop code. At the moment, getting a CloudBees subscription will not solve our problems as we are currently busy splitting the monolith.

Whether you should get a CloudBees membership really depends on your situation. Do you have a monolith? Has all code been untangled. Do you have microservices with clear boundaries or are they nanoservices? Long story short, do you have a robust architecture? If false, then CloudBees will also not be the holy grail.

While Googling, another post on stackoverflow was found that could be useful:

  The first difference is support (as others have mentioned). CloudBees
  offers enterprise grade support as well as a fully vetted and tested
  version of Jenkins that will be more stable under various plugins and
  deployments. You can actually purchase "Support Only" from CloudBees
  if you are satisfied with your OSS Jenkins deployment and simply want
  support during upgrades, patching, break/fix, etc.

  From a feature perspective, CloudBees brings a lot from an enterprise
  manageability, scalability, and security standpoint.

  Manageability: CloudBees comes with CJOC (CloudBees Jenkins Operations Center) built into the software. This is a single pane of
  glass management console that allows organizations or large teams to
  centrally manage the jenkins environment. Things like folders, RBAC,
  pipeline and master templates, and the ability to rapidly spin up/tear
  down a containerized jenkins master are all managed from this single
  console.
  Scaleability: CloudBees leverages Kubernetes to provide organizations with the ability to elastically scale Jenkins
  environments as needed. With CloudBees, your oganization can move away
  from a single "Monolithic"/"Frankenstein" master and into a
  multi-master and distributed pipeline architecture. This greatly
  reduces upgrade and administration complexity. This also eliminates
  the risk from a single point of failure that a monolithic architecture
  exposes.
  Security: CloudBees allows organizations to install Roll Based Access Control within Jenkins. This keeps users from accidentally or
  intentionally accessing repos that they shouldn't be allowed to
  interact with. CloudBees also provides "folders" to segregate specific
  job executions onto specific agents. Lastly CloudBees allows
  organizations to create pipeline templates and associated plugins for
  each team. These templates can be as rigid or loose as desired per the
  organizations security policies.

  CloudBees is regularly adding enhancements to further differentiate
  themselves from Jenkins Open Source and make themselves more appealing
  to large enterprise requirements.

  On top of the above, CloudBees has developed a presentation layer that
  rides on top of Jenkins for SDLC pipeline, CD monitoring, and metric
  tracking called DevOptics.


Is it possible to host an ECS cluster with p2 or other GPU instance types?

You can use custom instances of any type with ECS -- the key is running the ECS container agent on the instance of your choice. So, in general, I followed this documentation on that process with success -- http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-install.html -- but do note that you probably want to also specify the ECS cluster into which the instance goes (otherwise it will go to "default" cluster).

To specify the cluster you specify env var ECS_CLUSTER, typically in /etc/ecs/ecs.config prior to running the container agent. See here.


Kops: How to delete the multiple SSHPublicKey

kops delete secret --name cluster-1.dev.transein.com sshpublickey admin 2c:yy


how to stop and start kubernetes cluster

You can maintain your data while stopping the master if you have persistent volumes attached, this is especially the case when using GKE google Kubernetes engine. But if you wanted to keep your services running while completely stopping a master that would require High Availability, Allowing one master node to take over the responsibility of another master which is clustering.


Where is the apache2 filebeat module configured?

After looking a bit further, I found it in the sample called filebeat.full.yml

So I added this section to the filebeat.yml
Restarted filebeat

After running at 100% CPU for minute, Kibana is now immediately showing the results. That was easier than I expected.

#==========================  Modules configuration ============================
filebeat.modules:

#------------------------------- Apache2 Module ------------------------------
- module: apache2
  # Access logs
  access:
    #enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["MY-CUSTOM-PATH"]


Helm charts should not be built for a specific environment, IMHO. You can see that many of the official helm charts have a set of default values and they are configurable from outside. You also can create a helm chart like that and declare your deployment + environment-specific values with another tool like Helm Operator or ArgoCD. In this way, you can both simplify your build pipeline and increase dev/prod parity. You can use that one chart and promote the same build into different environments so you can be sure that different environments are identical except the configuration.
Example ArgoCD application manifests:
production-application.yaml:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: production-yourservice
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/yourorganisation/helm-charts
    path: yourservice
    targetRevision: master
    helm:
      values: |
        environment: production
        replicas: 10
  destination:
    server: https://kubernetes.default.svc
    namespace: production-yourservice

staging-application.yaml:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: staging-yourservice
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/yourorganisation/helm-charts
    path: yourservice
    targetRevision: master
    helm:
      values: |
        environment: staging
        replicas: 1
  destination:
    server: https://kubernetes.default.svc
    namespace: staging-yourservice


I think you could achieve something similar by first running Terraform against your RDS resources, and then running Terraform again on all resources.

On your first pass, you can use -target option to target only a subset of the resources. (ie. RDS resources).

Be aware that using this option is not recommended in Terraform docs. If I were you, I would use it for time being and change it as soon as one of these issues below is solved:
https://github.com/hashicorp/terraform/issues/10462
https://github.com/hashicorp/terraform/issues/2430
https://github.com/hashicorp/terraform/issues/4149

For details on how to use -target and its caveats, see: https://www.terraform.io/docs/commands/plan.html#resource-targeting


So what is happening is that the changed code just sat until internal testing was done (never a good thing as this is how code get stale quickly).  Once internal testing was ready, the code was then merged into the trunk causing it to be published into nugget under the trunk's feed.  Some business politics ensued and we finally got to a real single code base mindset from the product owner's perspective (which was the issue that got us into this situation).  Moving forward I don't anticipate this to arise.

Code can be branched out.  However, only code in the trunk will go to our internal nugget feed or our build/deployment processes.


AWS Setting instance to use domain from another account

If I'm getting this right; Maybe you can just create on your old account's Route53 public hosted zone a CNAME record (if you have a load balancer on your new account) or an A record for the instance on your new account. Just don't use the Alias functionality from Route53 as you wont find the resources from the new account.


How can I expose Ingress service on 80 and 443 to the outer world

It's would be better if you will use ClusterIP type as a service than apply Ingress resource on it
https://kubernetes.io/docs/concepts/services-networking/ingress/#single-service-ingress


Backing up an ephemeral Elasticsearch installation

ElasticSearch can backup itself when told to do so via an API call.
You just need to configure a destination, and optionally credentials, for example you make ElasticSearch create snapshots of its database in AWS S3.

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html

You can also automate these calls using a schedule with the Elastic curator tool.

https://www.elastic.co/guide/en/elasticsearch/client/curator/5.5/snapshot.html


New Kubernetes Cluster: remote error: tls: bad certificate

From the documentation you linked, it states,

  The referenced file must contain one or more certificates authorities to use to validate client certificates presented to the API server.

This means your client's certificate is invalid because either:

The server doesn't trust the client's signing certificate authority
The client doesn't trust the server's signing certificate authority
The certificate's DN doesn't match the hostname

The most likely scenario is that 1. The server doesn't trust the client's signing certificate authority since the server doesn't verify DNS for the client certificate and the error indicates this is a remote error not on the client. Most likely this means the client's certificate was signed on the client itself, not on the server.

You will need to make sure to generate the certificate on the client, have the server sign it and then transfer the certificate back to the client. Otherwise, you need to add the server's CA certificate to the Client's keystore.. You can ensure you do not have a DNS/DN mismatch by setting hosts file entries.


How can I add another module to my existing Terraform deployment plan?

Terraform is idempotent so will only change what needs to be changed.
So after applying your existing plan, if you add more modules or resources and run another plan and apply, it will add the new resources not the whole infrastructure.
Terraform import is used when you want to start managing a resource created outside your Terraform code for example created in the AWS console, AWS CLI).
It basically associates and existing resource with your new Terraform code, so that Terraform doesn't try to recreate it when you run the plan/ apply.


Git - Everyone Pushing to Same Branch in Different Files

This is not a "strategy". This is a sure way toward chaos.

The idea behind "trunk branch", which means having one branch for integration, does not mean that everyone should be pushing their changes to this branch directly. It only means that his branch is the place where integration happens.

Why do you even need two branches, Release and Master? What benefit does this bring? Probably just the benefit of confusing git history into an incomprehensible mess so that no one knows what is happening.

Stop this madness!

Make each change on its own pull-request branch (call it feature branches if you like).
Make sure that these pull-requests are not merged into the main integration branch before the code there goes through some minimal code-review.
Restrict from pushing directly to your integration branch.
For cleaner history, make sure that the pull-request branch if first rebased on top of integration.
Make sure that merges of pull-requests are not fast-forwarded and clearly visible in history.

The above will allow for several benefits,

Higher quality and less "human error" mistakes.
Learning and teaching each other cool dba tricks (during code reviews).
Keeping the integration branch green most of the time (it works!).
High trust that you are not shooting yourself in the leg using your process.
Way less "oops, I overwrote your change" kind of merges.
High trust that new team members are not going to mess things up.
Git history that is a thing of beauty and a joy forever.

And more ...

Prettier explanation of the "Feature Branch Workflow" is available on the Atlassian Git website here https://www.atlassian.com/git/tutorials/comparing-workflows/feature-branch-workflow


How to insert a minion Powershell variable into Salt Pillar or Mine?

You can only insert the return into the mine.

Just specify the cmd.run (or possibly cmd.script) in your mine_functions and you will get the whole return back.

If you want to have a script that sets up extra config on the minion, I would recommend writing a custom grain.

https://docs.saltstack.com/en/latest/topics/grains/#writing-grains


Artifacts mentioned in secondary-artifacts section not getting uploaded to s3 bucket

In the CodeBuild "Artifacts" configuration, there is an "Add Artifact" button.

You need to click that and reference your secondary artifact to get it to upload to S3 as well.


I think it depends on the context.
I feel your pain with regard to needing to tinker and tweak a MySQL/MariaDB instance. It's not extremely difficult to get an instance up and running and THEN populate the DB from, say, a mysqldump by attaching volumes to the MySQL instance either using a Dockerfile for the instance or by "manually" bash-ing into it and loading your tables, tweaking your network config, setting up complex user permissions, etc.
But it's usually a STRUGGLE to deploy a "ready to go" mysql instance from a docker-compose.yml in my initial experience using Docker Composer for a MySQL container in an app deployment, but it IS doable.
What kind of scale are you trying to achieve? Do you NEED Kubernetes or Docker Swarm? I think in the case of most apps, the answer is honestly no. You can almost always come up with a crafty workaround with ba(sh) scripts or RUN/CMD (Dockerfile) or command: (docker-compose).
Are you trying to support an automated deployment? or just get an app up and running? or trying to increase resilience? Increase capacity at scale?


Is there a good standard to follow when choosing a service mesh for your microservice application?

I would say that Istio is the de facto standard for Service Mesh. It is launched by both Google and IBM.

Welcome to the service mesh era: Introducing a new Istio blog post series
IBM, Google Cloud and the open community launch Istio 1.0 to bring microservices to the enterprise


Where is Puppet's enviroments directory on Amazon Linux2?

You tried look on /opt or /etc probably the puppetlabs folder must be in one of these directories.
Amazon Linux it's based on CentOS/RedHat family.


Configuration Management Tools and Rollbacks / Approaching Immutable Infrastructure

You hint in the title about immutable infrastructure, so it sounds like you already know the solution:  don't change existing servers, but bring up new ones with your changes, switch over to them, and switch back to the old ones if necessary.

Theoretically any of the standard config management tools can roll back by simply checking out a previous commit and rerunning the config.  However, practically speaking this often isn't the case, as you may have introduced a new config rule (and so whatever it is doing is simply ignored in older versions) or you've written a rule that isn't easily reversible (for instance, does some destructive data operation).  If you were very careful in how you always write your playbooks, you could avoid this, but a few less error-prone solution is just to use an immutable infrastructure approach.


IMHO that depends on what the role of the tester was before such transformation. BTW, I believe my answer applies to the DevOps transformation in general, not only to the You build it, you run it paradigm.

If the role was a drone job - mindlessly executing tests - it's bound to go away, automation will eat such jobs.

If the role included writing the test plans and specifications they could continue to do that or even morph into writing the corresponding automation scripts, side-by-side with the developers. Or standalone - in some environments it is mandatory to use a completely separate methodology/infrastructure/personnel for testing than the one(s) used for development. In many organisations testers are software engineers, just as developers.

If the role was executing intentionally unreliable/surprise tests, specifically requiring human traits (emotion, subjectivity, inspiration, reflexes, intuition, etc) to complement automated testing - that also won't change.


Does `npm audit` add any value when using `dependabot`

Dependabot and npm audit both poll the Node Security Working Group database for Node-based projects. However, Dependabot has the added ability to check dependencies in numerous other types of projects as well.
Also, each report Dependabot generates includes useful info and links directly to a GitHub Advisory Database listing (e.g., CVE-2017-16021) that itself has multiple links to other listings such as NIST. On the other hand, npm audit's reporting is limited to a single link to an npm advisory listing with far less info (e.g., #100).
Overall, Dependabot covers all the bases — and more — that npm audit does. No need to use both unless you need a customized audit flow in your package scripts.


Why does Docker hub allow non-official Docker images and why do they exist?

As with most elements of Information Security, there is a balancing act between usability and security; an increase in usability often results in a correlated decrease in security and visa versa.

As a thought experiment...

  I can make any server 100% secure by wiping the hard drives, putting it through a shredder and then melting down all of the components parts.  It also makes it 0% usable.

Docker Hub is no different, the desire is to make these images accessible for the common good, at the cost of traceability and provenance of the images.  Fortunately, there are several tools such as JFrog Artifactory and Sonatype Nexus Lifecycle which enable you to apply governance to any Open Source repository (i.e. Docker Hub, Nuget, NPM, etc).

One aspect that I think Docker Hub gets wrong, that other sites get right, namely NPM, is presenting clear information about what Source Repository an image is generated from, that I suspect is due to the decoupled nature of Docker Images from the source code that generates them.


running a phpUnit test within a docker container

Don't quote the command you're trying to run, just do:

docker run --rm -it -v ~/Users/mow/Documents/devFolder/testdev:/app testdev_php php ./vendor/bin/phpunit

Your image is looking for a file called "php ./vendor/bin/phpunit" instead of looking for "php" and passing everything else on the line to it.

This is the real reason docker requires all your parameters before the image, because the parameter after the image is the command and everything after it is parameters to that command - don't quote everything.

The Dockerfile equivalent is:

ENTRYPOINT ["/init","parameter1","parameter2","parameter3"]

If it still doesn't work, check the parent Dockerfiles for the location of "php"

docker run --rm -it -v ~/Users/mow/Documents/devFolder/testdev:/app testdev_php /usr/local/bin/php ./vendor/bin/phpunit


Does active choice reactive parameter support hide fields option?

No, this is not possible, because the number and type of parameters for a build are fixed and cannot be modified on-the-fly.  The only way to modify which parameters show up in the "build with parameters" UI is to change the job configuration, which alters the parameters permanently.

In my opinion, if you need to do several different tasks which require several different input parameters (such as "build", "test", "install", "deploy", etc.), you should have separate jobs in Jenkins for each task.  I personally think it is a good ideal in general to avoid complex Jenkins jobs which require a lot of input parameters and try instead to simplify job configuration in order to reduce the likelihood of subtle, difficult-to-debug errors when executing jobs.


Chef - assign a value to variable in execution or converge phase

You have a problem of compile vs converge times as you guessed.

You code between the execute resource and the ruby_block resource is executed before the execute resource is converged.

The easiest way to circumvent that is to put that code within a ruby_block, and to avoid scoping problem use node.run_state hash to store the value to be checked.

A rewrite of your recipe which should work would be:

#
# Cookbook:: test_cookbook
# Recipe:: check-vpn-ip
#
# Copyright:: 2019, The Authors, All Rights Reserved.

#Getting the IP address using the ruby's Socket class.

ip_list = Socket.ip_address_list
vpn_ip_list = ip_list.select{ |ip| ip.ip_address.match(/^10.12/) }
!vpn_ip_list.empty? ? ip_addr = vpn_ip_list.first.ip_address : ip_addr = ""

execute 'manually_start_open_vpn' do
  command "sudo openvpn #{node['openvpn-conf-path']}/#{host}.conf &"
  action :nothing
  only_if {ip_addr.length.eql?(0)}
end

ruby_block 'check_vpn_ip_list' do
  block do
    new_ip_list = Socket.ip_address_list
    new_vpn_ip_list = new_ip_list.select{ |ip| ip.ip_address.match(/^10.12/) }
    node.run_state['newvpn_ip_addr'] = !new_vpn_ip_list.empty? ?  new_vpn_ip_list.first.ip_address : ""
  end
end

ruby_block 'chat-bot' do
  block do
    machine_data = {text: "OpenVPN IP not assigned to #{host}
        software_version: 18.4.4 \n This is a test message please
        ignore @all"}.to_json
    header = {'Content-Type': 'text/json'}
    http = Net::HTTP.new(google_chat_uri.host,                       google_chat_uri.port)
    http.use_ssl = true
    request = Net::HTTP::Post.new(google_chat_uri.request_uri, header)
    request.body = machine_data
    response = http.request(request)
  end
  only_if {node.run_state['newvpn_ip_addr'].length.eql?(0)}
  action :nothing
end

There's no real use of response = in the last line as you don't use it later, but I let it as is in case you wish to log in the ruby block the response.


Preserving ownership of a folder after mounting it

At least in the *nix world the ownership (i.e. uid and gid) of a file/folder on a mounted filesystem doesn't change, it is the actual ownership from the host exporting that filesystem (if ownership is supported by that filesystem).

Note that if the container has different user/group names matching the uid/gid of the file/folder they will appear under the container's names in ls -l, not as the host's names (or as numbers if no container names match the respective IDs).

So if on the host machine the folder is owned by root, it will also be owned by root in the container (in other words it's nothing "special" about it being owned by root vs other users). The only way to allow the container app to write to the filesystem is to  change the ownership accordingly on the host system (or give it other write permissions).

Maybe worth noting that if the host's filesystem itself doesn't support ownership, it can still be "faked" inside the container using the uid and gid mount options (in the *nix world). See Ownership of a mount point.


How to only include directory contents in Azure Publish Artifact without full filepath?

In the copy files task, there is an optional parameter flattenFiles (bool). This does the job.


Is "running several Linux distributions on the same host" a good justification for container adoption?

That is not a good argument and it shows a basic misunderstanding of what containers are and how they should be used. Containers are NOT virtual machines.

A container runs a (short lived) process. Think executable, like ls, or top, or a shell or Python script. When the process ends, the container is destroyed. The process is isolated from other processes running on the same machine. Basically the kernel reserves some space (memory/CPU) for that process, runs the executable inside that isolated space, then releases the reservation. The reserved space is the container.

A virtual machine on the other hand is an emulator. It pretends there is an entire computer and it needs an operating system to, well, operate it. It boots a kernel, starts an init system and then a whole bunch of processes to deal with every aspect of the emulated hardware.

A container is supposed to be small and fast. And it's not generally supposed to run for days, or even hours. Compared to a virtual machine, which gives the user more freedom, but also takes more resources. Packing an entire distribution like Debian or RedHat inside a container would defeat the purpose.

For example, you're not supposed to run a web server (say, Nginx + PHP) inside a container. When a page is requested, you run the PHP script inside a container to generate the page, hand the output over to Nginx running outside, then the container is destroyed. If you want Nginx+PHP in the same space, you're better off with a virtual machine. There may be reasons to set it up as a container, but technically speaking Nginx workers already do the work of containers, so you'd just be duplicating that.


In your job, I noticed the below curl command, where two things need to be corrected:

token=$CI_JOB_TOKEN - this TOKEN should be generated in project B under Settings -> CI/CD -> Pipeline triggers (Add triggers)
"ref=$CI_COMMIT_REF_NAME" - this is pointing to a branch with name $CI_COMMIT_REF_NAME on project B and this will only execute if it find a project with that branch name, so if you are trying to trigger it on master branch or any specific branch, you should replace it with something like -F "ref=master" in your curl command

 - curl -X POST -F "token=$CI_JOB_TOKEN" -F "ref=$CI_COMMIT_REF_NAME" https://gitlab.com/api/v4/projects/1234/trigger/pipeline

I use the same trigger with few variables to execute one of my job in another project

- "curl -k -X POST -F token=$PROJECT_B_TOKEN -F ref=master -F variables[PROJECT_NAME]=$CI_PROJECT_NAME -F variables[TAG]=$CI_BUILD_REF_NAME -F variables[SOME_VARIABLE_IDENTIFIER]=DevOps -F variables[USER]=$GITLAB_USER_LOGIN https://gitlab.mycompany.com/api/v4/projects/1234/trigger/pipeline"

The person who created in token in project B, needs to be owner in project A to execute this trigger, else it fails with a 404 message.


How to build, package and deploy a react-native-web app in a Docker container?

It's a pretty general question, I will try to answer the best I can:
If you want to just build and run an image:
Build the project: npx expo-cli init react_native_app --npm

You need to create a Dockerfile file in your repository
Docker build -t {tag_name} . (You can replace '.' eith the Dockerfile path)
Docker run...
If you want you can create docker compose file to better control your app.
Reference:
https://www.rockyourcode.com/how-to-run-react-native-expo-web-in-a-docker-container/


What's the difference between DevOps and Automation?

A big part of DevOps is making it possible to release very often. That comes with automated build, automated testing, etc. You can say that to achieve its goals, DevOps need to use automation to be efficient.

Here's how DevOps and automation are related. DevOps is not just automation, there's more to it. Conversely, automation is not exclusively used by "DevOps people". A lot of automation was taking place in IT before DevOps came around.

Please don't consider to above diagram to represent all that is DevOps, or all this is automation. It is to help the reader picture how the two concepts are related.


How can I unify my distributed build environments using Jenkins and Docker?

It sounds like you have a main Jenkins orchestration server and six additional nodes added for build capacity. If you are attempting to improve scalability, there are quite a few options.
If you have containers representing the build environments (A, B, C D), an option would be docker swarm across the existing machines, allowing all of the hosts to be managed as a group and your containers to be dropped on them.
https://github.com/jenkinsci/docker-swarm-plugin
Similarly, but more complicated, create a kubernetes cluster (Openshift, Rancher, etc.) with your current six machines and allow the main Jenkins to spin up and use containers on the clustered kubernetes as needed. This would allow you to create, add additional machines to the cluster, if scaling is needed. You could even add your main Jenkins server to the cluster and run Jenkins as a container.
https://github.com/jenkinsci/kubernetes-plugin
Another option is to add cloud (GCP, Azure, AWS) capacity as a node(s) to push overflow to pay as you need capacity in the cloud.
https://github.com/jenkinsci/google-compute-engine-plugin
Lastly, you could also use your current solution, as is, while developing a cloud native CI/CD pipeline and slowly migrate to a more modular, scaling infrastructure. There are so many options in this space, some not even needing Jenkins, solutions could be tailored to specific needs.
I hope this gives you some ideas and hints!